Nov 01 06:09:06.382822 ip-10-0-138-21 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 06:09:06.836502 ip-10-0-138-21 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 06:09:06.836502 ip-10-0-138-21 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 06:09:06.836502 ip-10-0-138-21 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 06:09:06.836502 ip-10-0-138-21 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 06:09:06.841058 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.840979    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.841058 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841059    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841062    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841067    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841071    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841073    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841076    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841079    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841081    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841084    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841086    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841089    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841092    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841094    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841104    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841107    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841110    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841112    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841115    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841118    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841120    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.841130 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841123    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841125    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841128    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841130    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841133    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841136    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841139    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841144    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841150    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841153    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841155    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841158    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841162    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841166    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841169    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841172    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841198    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841201    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841205    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.841606 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841209    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841212    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841215    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841218    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841221    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841223    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841226    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841229    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841231    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841233    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841236    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841238    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841241    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841243    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841246    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841249    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841252    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841256    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841259    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841262    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.842042 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841264    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841267    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841270    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841272    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841275    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841278    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841280    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841283    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841285    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841288    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841290    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841293    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841295    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841298    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841301    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841303    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841306    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841308    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841311    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841314    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.842497 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841316    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841318    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841321    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841324    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841326    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841328    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841332    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841335    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841338    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841340    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841343    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841345    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841348    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841352    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841354    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841357    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841359    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841362    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841364    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.842973 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841367    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841369    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841372    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841374    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841377    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841379    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.841382    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842286    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842294    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842297    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842300    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842303    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842306    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842309    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842311    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842314    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842317    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842319    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842322    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.843411 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842324    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842327    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842330    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842332    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842335    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842337    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842340    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842342    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842345    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842347    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842350    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842352    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842354    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842357    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842359    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842362    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842364    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842366    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842369    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842371    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.844050 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842374    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842382    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842384    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842388    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842390    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842393    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842396    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842398    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842401    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842403    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842406    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842408    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842411    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842414    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842416    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842419    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842422    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842424    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842427    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842429    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.844703 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842431    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842434    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842436    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842439    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842442    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842444    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842447    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842450    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842452    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842455    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842457    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842460    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842462    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842466    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842476    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842479    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842483    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842485    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842488    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.845186 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842490    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842493    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842495    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842498    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842501    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842503    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842506    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842508    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842511    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842515    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842519    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842522    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842525    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842550    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842554    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842557    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842559    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842563    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842565    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.845662 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842568    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842571    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842573    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842576    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842578    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842581    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842583    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842587    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842590    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842598    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842601    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842604    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842606    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842609    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842611    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.842614    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843244    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843250    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843253    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843257    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.846095 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843260    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843262    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843266    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843268    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843271    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843273    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843276    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843278    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843281    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843283    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843286    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843289    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843291    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843294    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843297    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843299    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843302    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843305    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843308    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843310    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.846555 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843313    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843319    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843322    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843324    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843326    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843329    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843331    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843334    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843336    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843339    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843342    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843344    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843347    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843349    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843351    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843354    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843357    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843361    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843364    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.847034 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843366    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843369    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843373    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843376    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843379    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843382    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843384    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843387    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843389    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843392    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843395    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843397    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843400    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843402    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843405    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843408    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843411    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843413    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843416    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.847475 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843418    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843422    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843424    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843427    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843429    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843432    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843434    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843436    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843439    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843441    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843444    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843447    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843449    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843452    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843454    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843458    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843460    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843463    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843467    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843471    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.847970 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843474    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843477    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843479    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843482    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843485    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843487    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843490    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843493    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843495    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843555    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843768    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843775    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843779    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843784    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843788    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843795    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843800    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843805    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843810    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843814    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.848435 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843825    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843830    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843834    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.843839    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.844882    2572 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.844946    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845080    2572 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845091    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845098    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845102    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845107    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845111    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845115    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845118    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845121    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845125    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845128    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845131    2572 flags.go:64] FLAG: --cgroup-root=""
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845134    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845137    2572 flags.go:64] FLAG: --client-ca-file=""
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845140    2572 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845142    2572 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845148    2572 flags.go:64] FLAG: --cluster-domain=""
Nov 01 06:09:06.848996 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845152    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845155    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845158    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845162    2572 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845174    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845177    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845180    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845183    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845186    2572 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845190    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845193    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845196    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845199    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845205    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845209    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845212    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845215    2572 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845218    2572 flags.go:64] FLAG: --enable-server="true"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845221    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845227    2572 flags.go:64] FLAG: --event-burst="100"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845230    2572 flags.go:64] FLAG: --event-qps="50"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845233    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845237    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845240    2572 flags.go:64] FLAG: --eviction-hard=""
Nov 01 06:09:06.849598 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845244    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845247    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845250    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845253    2572 flags.go:64] FLAG: --eviction-soft=""
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845256    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845259    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845262    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845265    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845268    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845272    2572 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845275    2572 flags.go:64] FLAG: --feature-gates=""
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845279    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845282    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845285    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845288    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845291    2572 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845294    2572 flags.go:64] FLAG: --help="false"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845297    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845301    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845304    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845306    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845311    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845315    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 06:09:06.850169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845319    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845322    2572 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845325    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845327    2572 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845330    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845334    2572 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845337    2572 flags.go:64] FLAG: --kube-reserved=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845340    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845343    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845346    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845349    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845352    2572 flags.go:64] FLAG: --lock-file=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845355    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845358    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845361    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845367    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845370    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845373    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845376    2572 flags.go:64] FLAG: --logging-format="text"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845379    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845382    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845385    2572 flags.go:64] FLAG: --manifest-url=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845388    2572 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845393    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845396    2572 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 06:09:06.850698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845400    2572 flags.go:64] FLAG: --max-pods="110"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845403    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845407    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845410    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845413    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845418    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845421    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845425    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845437    2572 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845440    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845443    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845446    2572 flags.go:64] FLAG: --pod-cidr=""
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845450    2572 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845453    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845456    2572 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845459    2572 flags.go:64] FLAG: --port="10250"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845462    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845465    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-066853833f9932c8e"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845468    2572 flags.go:64] FLAG: --qos-reserved=""
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845471    2572 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845474    2572 flags.go:64] FLAG: --register-node="true"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845477    2572 flags.go:64] FLAG: --register-with-taints=""
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845481    2572 flags.go:64] FLAG: --registry-burst="10"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845484    2572 flags.go:64] FLAG: --registry-qps="5"
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845487    2572 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845490    2572 flags.go:64] FLAG: --reserved-memory=""
Nov 01 06:09:06.851286 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845494    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845497    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845500    2572 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845503    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845506    2572 flags.go:64] FLAG: --runonce="false"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845508    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845512    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845514    2572 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845517    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845520    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845523    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845528    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845543    2572 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845546    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845549    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845552    2572 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845554    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845559    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845569    2572 flags.go:64] FLAG: --system-cgroups=""
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845573    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845581    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845584    2572 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845586    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845591    2572 flags.go:64] FLAG: --tls-min-version=""
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845594    2572 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 06:09:06.852043 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845597    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845600    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845603    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845606    2572 flags.go:64] FLAG: --v="2"
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845610    2572 flags.go:64] FLAG: --version="false"
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845614    2572 flags.go:64] FLAG: --vmodule=""
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845618    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.845622    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845770    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845774    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845778    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845780    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845784    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845787    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845789    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845792    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845795    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845797    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845802    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845804    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.853093 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845807    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845812    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845815    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845818    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845820    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845824    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845826    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845829    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845831    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845834    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845836    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845839    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845841    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845844    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845846    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845849    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845851    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845854    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845856    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.854001 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845861    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845865    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845868    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845872    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845874    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845877    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845880    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845882    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845885    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845887    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845890    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845894    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845896    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845900    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845902    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845905    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845908    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845910    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845912    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845915    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.854814 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845918    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845921    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845923    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845926    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845928    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845931    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845933    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845935    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845938    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845940    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845943    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845945    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845948    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845950    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845953    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845956    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845958    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845961    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845963    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845966    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.855693 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845968    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845971    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845973    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845977    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845981    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845984    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845986    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845989    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845991    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845994    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845996    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.845999    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846003    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846006    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846008    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846011    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846014    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846017    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846019    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846022    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.856684 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846024    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846027    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846030    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846032    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846035    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846038    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846040    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846043    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846045    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846048    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846051    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846053    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846056    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846058    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846061    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.846090    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 06:09:06.857211 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846128    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846133    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846136    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846139    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846141    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846145    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846147    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846150    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846152    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846155    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846158    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846161    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846163    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846166    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846169    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846171    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846174    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846176    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846179    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.857812 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846182    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846184    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846187    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846190    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846194    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846196    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846199    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846202    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846205    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846208    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846210    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846213    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846216    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846220    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846223    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846226    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846228    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846231    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846233    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.858618 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846236    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846238    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846243    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846247    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846251    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846254    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846256    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846259    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846262    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846264    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846267    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846269    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846272    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846275    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846278    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846280    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846283    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846285    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846288    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846290    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.859171 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846293    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846295    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846298    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846301    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846303    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846306    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846311    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846314    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846316    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846319    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846322    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846324    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846327    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846329    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846332    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846334    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846337    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846340    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846342    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846345    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.859740 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846347    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846350    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846352    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846355    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846358    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846360    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846363    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846365    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846368    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846370    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846373    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846375    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846378    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846380    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846383    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846385    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846388    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846391    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846396    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846399    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.860214 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846402    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846404    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846407    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846409    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846412    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846414    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846417    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.846420    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.854583    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.854602    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.854628    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.854635    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854718    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854725    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854731    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854736    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854740    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854744    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854749    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854753    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.860698 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854757    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854761    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854765    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854769    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854774    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854778    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854782    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854787    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854791    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854795    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854799    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854803    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854808    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854813    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854818    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854822    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854829    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854834    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854839    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.861249 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854845    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854850    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854856    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854862    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854867    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854871    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854875    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854879    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854884    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854889    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854893    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854898    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854902    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854907    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854911    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854916    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854920    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854925    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854929    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.861790 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854934    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854938    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854942    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854946    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854950    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854955    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854959    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854963    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854967    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854971    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854976    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854980    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854985    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854989    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854994    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.854999    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855003    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855008    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855013    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855017    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.862288 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855021    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855026    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855030    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855035    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855039    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855043    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855047    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855052    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855056    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855061    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855065    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855069    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855074    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855078    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855082    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855086    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855091    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855095    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855100    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855104    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.862773 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855109    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855113    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855117    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855121    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855126    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855130    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855136    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855142    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855146    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855151    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855157    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855161    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855165    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855170    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855174    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855179    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855183    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855187    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855192    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.863292 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855197    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.855237    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855463    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855471    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855477    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855482    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855487    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855491    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855495    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855500    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855504    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855511    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855516    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855521    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855525    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855529    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.863775 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855553    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855557    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855562    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855566    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855570    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855574    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855578    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855582    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855588    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855592    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855596    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855601    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855605    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855609    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855613    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855618    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855622    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855626    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855630    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855634    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.864213 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855638    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855642    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855646    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855650    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855654    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855659    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855663    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855667    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855671    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855675    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855679    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855683    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855687    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855691    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855695    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855699    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855703    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855707    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855711    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.864849 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855717    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855724    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855731    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855736    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855740    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855744    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855748    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855752    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855756    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855760    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855765    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855769    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855773    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855777    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855781    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855785    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855789    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855793    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855797    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855802    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855806    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.865284 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855811    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855815    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855819    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855823    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855827    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855831    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855836    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855840    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855844    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855848    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855852    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855856    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855861    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855864    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855869    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855873    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855878    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855882    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855886    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.865926 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855890    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855894    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855898    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855902    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855906    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855910    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855915    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855919    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855923    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855926    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855930    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855935    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.855939    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.855978    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856031    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:06.866368 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856038    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856043    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856047    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856052    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856056    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856060    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856065    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856069    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856073    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856077    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856081    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856085    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856089    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856095    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856099    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856103    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856107    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856111    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856116    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:06.866735 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856120    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856124    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856127    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856131    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856136    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856140    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856144    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856151    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856156    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856161    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856165    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856170    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856174    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856182    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856194    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856199    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856204    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856208    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856212    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856216    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:06.867177 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856220    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856225    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856229    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856233    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856237    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856241    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856246    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856251    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856256    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856259    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856264    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856268    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856272    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856277    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856281    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856285    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856289    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856294    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856298    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:06.867653 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856302    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856306    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856310    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856315    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856319    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856324    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856330    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856334    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856339    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856343    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856347    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856351    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856355    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856361    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856365    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856370    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856374    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856378    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:06.868085 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856382    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856386    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856390    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856397    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856402    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856406    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856410    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856415    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856420    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856424    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856428    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856432    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856439    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856444    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856450    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856454    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856458    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856462    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856466    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856471    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:06.868526 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856476    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856481    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856486    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856490    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856494    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856498    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856503    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856507    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:06.856511    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.856604    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.861197    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.862181    2572 server.go:1014] "Starting client certificate rotation"
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.862293    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.862327    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.864655    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 06:09:06.869067 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.865763    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 06:09:06.875665 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.875646    2572 log.go:25] "Validated CRI v1 runtime API"
Nov 01 06:09:06.879155 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.879141    2572 log.go:25] "Validated CRI v1 image API"
Nov 01 06:09:06.879914 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.879888    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 06:09:06.883818 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.883799    2572 fs.go:135] Filesystem UUIDs: map[42e24511-0162-4549-8f31-a532e06c9607:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 94f79eab-d256-4b5d-92a0-0b07450f5f63:/dev/nvme0n1p4]
Nov 01 06:09:06.883864 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.883819    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 06:09:06.888775 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.888663    2572 manager.go:217] Machine: {Timestamp:2026-11-01 06:09:06.887475301 +0000 UTC m=+0.372521653 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3059249 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27820165c162773ee030d57fda1962 SystemUUID:ec278201-65c1-6277-3ee0-30d57fda1962 BootID:916f0823-affe-4c60-8a5c-f801ce61d38a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:07:da:3f:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:07:da:3f:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:e9:d4:e6:ec:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 06:09:06.888775 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.888775    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 06:09:06.888895 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.888854    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 06:09:06.892702 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.892677    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 06:09:06.892830 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.892704    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-21.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 06:09:06.892874 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.892833    2572 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 06:09:06.892874 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.892840    2572 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 06:09:06.892874 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.892850    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 06:09:06.893579 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.893561    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 06:09:06.894578 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.894568    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 06:09:06.906175 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.906160    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 06:09:06.906304 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.906293    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 06:09:06.907110 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.907090    2572 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 06:09:06.907180 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.907119    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 06:09:06.907180 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.907133    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 06:09:06.907180 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.907146    2572 kubelet.go:405] "Adding apiserver pod source"
Nov 01 06:09:06.907180 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.907158    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 06:09:06.907559 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.907527    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 06:09:06.909794 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.909776    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 06:09:06.915470 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.915457    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 06:09:06.916214 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916201    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 06:09:06.916256 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916233    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 06:09:06.916377 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916367    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 06:09:06.916410 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916383    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 06:09:06.916410 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916390    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 06:09:06.916410 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916396    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 06:09:06.916410 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916402    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 06:09:06.916410 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916407    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 06:09:06.916912 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916903    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 06:09:06.916942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916913    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 06:09:06.916942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916920    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 06:09:06.916942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916926    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 06:09:06.916942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916934    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 06:09:06.916942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.916943    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 06:09:06.917660 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.917651    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 06:09:06.917692 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.917661    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 06:09:06.921351 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.921339    2572 server.go:1263] "Started kubelet"
Nov 01 06:09:06.921443 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.921416    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 06:09:06.922015 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.921973    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 06:09:06.922070 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.922033    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 06:09:06.922803 ip-10-0-138-21 systemd[1]: Started Kubernetes Kubelet.
Nov 01 06:09:06.925382 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.925361    2572 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 06:09:06.925885 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:06.925868    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 06:09:06.926676 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.926651    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 06:09:06.931250 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.931232    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 06:09:06.931988 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.931714    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 06:09:06.931988 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:06.931817    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:06.932550 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932511    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 06:09:06.932621 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932553    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 06:09:06.932672 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932652    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 06:09:06.932732 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932683    2572 factory.go:55] Registering systemd factory
Nov 01 06:09:06.932870 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932803    2572 factory.go:223] Registration of the systemd container factory successfully
Nov 01 06:09:06.932927 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932706    2572 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 06:09:06.932927 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.932920    2572 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 06:09:06.933118 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.933103    2572 factory.go:153] Registering CRI-O factory
Nov 01 06:09:06.933181 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.933121    2572 factory.go:223] Registration of the crio container factory successfully
Nov 01 06:09:06.933181 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.933170    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 06:09:06.933279 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.933187    2572 factory.go:103] Registering Raw factory
Nov 01 06:09:06.933279 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.933203    2572 manager.go:1199] Started watching for new ooms in manager
Nov 01 06:09:06.934225 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.934209    2572 manager.go:319] Starting recovery of all containers
Nov 01 06:09:06.944096 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.944079    2572 manager.go:324] Recovery completed
Nov 01 06:09:06.948605 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.948590    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:06.950375 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950351    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:06.950433 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950387    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:06.950433 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950398    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:06.950956 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950930    2572 cpu_manager.go:226] "Starting" policy="none"
Nov 01 06:09:06.950956 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950951    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:06.951094 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950955    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 06:09:06.951094 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.950988    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 06:09:06.951612 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.951597    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:06.951695 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.951628    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:06.951695 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.951644    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:06.953090 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.953078    2572 policy_none.go:50] "Start"
Nov 01 06:09:06.953148 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.953094    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 06:09:06.953148 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.953103    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 06:09:06.955117 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.955107    2572 policy_none.go:44] "Start"
Nov 01 06:09:06.985164 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.985130    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 06:09:06.987480 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.987464    2572 manager.go:341] "Starting Device Plugin manager"
Nov 01 06:09:06.987576 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:06.987501    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 06:09:06.987576 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.987513    2572 server.go:87] "Starting device plugin registration server"
Nov 01 06:09:06.991723 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.991423    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 06:09:06.991723 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.991565    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 06:09:06.991723 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.991657    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.991740    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:06.991749    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:06.996276    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:06.996326    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.010962    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.012361    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.012387    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.012407    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 06:09:07.017083 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.012438    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 06:09:07.065028 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.064952    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-21.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 06:09:07.065028 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.064967    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-21.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 06:09:07.067141 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.064958    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-21.us-west-1.compute.internal.18e3d6e2f72ab59f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:06.921313695 +0000 UTC m=+0.406360044,LastTimestamp:2026-11-01 06:09:06.921313695 +0000 UTC m=+0.406360044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.us-west-1.compute.internal,}"
Nov 01 06:09:07.092460 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.092436    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:07.093289 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.093272    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:07.093373 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.093303    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:07.093373 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.093314    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:07.093373 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.093340    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.112926 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.112882    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"]
Nov 01 06:09:07.113044 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.112959    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:07.113817 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.113802    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:07.113884 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.113829    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:07.113884 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.113839    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:07.115362 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.115350    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:07.115504 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.115489    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.115563 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.115521    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:07.116033 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.116019    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:07.116096 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.116044    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:07.116096 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.116066    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:07.116096 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.116078    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:07.116185 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.116046    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:07.116185 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.116126    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:07.117667 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.117652    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.117667 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.117676    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:07.118390 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.118374    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:07.118475 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.118406    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:07.118475 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.118422    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:07.133619 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.133597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.133692 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.133625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.133692 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.133642    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a65f0fc8a51566f4a4d62ed9c49d84d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"5a65f0fc8a51566f4a4d62ed9c49d84d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.135151 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.135134    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 06:09:07.135194 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.135189    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nw55b"
Nov 01 06:09:07.146010 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.145989    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.149136 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.149120    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.167843 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.167809    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 06:09:07.167951 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.167881    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-21.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 06:09:07.167951 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.167890    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 06:09:07.167951 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.167833    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-21.us-west-1.compute.internal.18e3d6e2f8e62309  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-21.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:06.950374153 +0000 UTC m=+0.435420503,LastTimestamp:2026-11-01 06:09:06.950374153 +0000 UTC m=+0.435420503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.us-west-1.compute.internal,}"
Nov 01 06:09:07.168093 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.168039    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 06:09:07.221068 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.221038    2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-21.us-west-1.compute.internal\" not found" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.234234 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.234208    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a65f0fc8a51566f4a4d62ed9c49d84d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"5a65f0fc8a51566f4a4d62ed9c49d84d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.234234 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.234236    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.234387 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.234270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.234387 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.234323    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a65f0fc8a51566f4a4d62ed9c49d84d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"5a65f0fc8a51566f4a4d62ed9c49d84d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.234387 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.234325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.234387 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.234325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.368256 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.368174    2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-21.us-west-1.compute.internal\" not found" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.421875 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.421843    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:07.422904 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.422887    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:07.422986 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.422920    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:07.422986 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.422932    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:07.422986 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.422964    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.449807 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.449778    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.452636 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.452618    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.478709 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.478686    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:07.478787 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.478713    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-21.us-west-1.compute.internal\": node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:07.591653 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.591576    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:07.691942 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.691859    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:07.792924 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.792899    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:07.862095 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:07.862059    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 06:09:07.862570 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.862263    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-21.us-west-1.compute.internal.18e3d6e2f8e66a0a\": read tcp 10.0.138.21:55978->3.140.212.52:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-21.us-west-1.compute.internal.18e3d6e2f8e66a0a  default   4999 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-21.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-21.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-21.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:06 +0000 UTC,LastTimestamp:2026-11-01 06:09:07.113834039 +0000 UTC m=+0.598880390,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-21.us-west-1.compute.internal,}"
Nov 01 06:09:07.893403 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.893372    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:07.994322 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:07.994262    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:08.094890 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:08.094851    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:08.113657 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.113636    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:08.114608 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.114589    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:08.114688 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.114624    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:08.114688 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.114640    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:08.140103 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.140068    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 06:04:08 +0000 UTC" deadline="2028-07-29 09:07:01.471932614 +0000 UTC"
Nov 01 06:09:08.140103 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.140099    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15266h57m53.331836949s"
Nov 01 06:09:08.195891 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:08.195859    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:08.205753 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:08.205733    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:08.296228 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:08.296149    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:08.397224 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:08.397191    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:09.646766 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:09.646743    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:09.658549 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:09.658515    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d718e8dfa8061bd8b70748a702b624.slice/crio-a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3 WatchSource:0}: Error finding container a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3: Status 404 returned error can't find the container with id a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3
Nov 01 06:09:09.658855 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:09.658834    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a65f0fc8a51566f4a4d62ed9c49d84d.slice/crio-dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42 WatchSource:0}: Error finding container dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42: Status 404 returned error can't find the container with id dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42
Nov 01 06:09:09.664077 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:09.664062    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:09:09.747455 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:09.747360    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:09.847822 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:09.847792    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:09.876361 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:09.876326    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:09.876527 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:09.876504    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:09.916074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:09.916048    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:09.982378 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:09.982345    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:10.038844 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.038794    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 06:09:10.041784 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.041770    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:10.056880 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.056862    2572 apiserver.go:52] "Watching apiserver"
Nov 01 06:09:10.081066 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.081041    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 06:09:10.100010 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.099984    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 06:09:10.158719 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.158691    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 06:09:10.158985 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.158968    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-v48xf","kube-system/konnectivity-agent-w6l2n","openshift-multus/multus-additional-cni-plugins-bq58b","openshift-multus/network-metrics-daemon-l6492","openshift-image-registry/node-ca-r5fnn","openshift-multus/multus-pvbrt","openshift-network-diagnostics/network-check-target-7llbh","openshift-network-operator/iptables-alerter-pr9kp","kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-2rnnc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q","openshift-cluster-node-tuning-operator/tuned-8rjvr","openshift-dns/node-resolver-sdt2h"]
Nov 01 06:09:10.162698 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.162679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:10.162796 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.162772    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:10.164943 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.164926    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.167096 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.167073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.169459 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.169440    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.169551 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.169518    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:10.171546 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.171531    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.173750 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.173735    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.175815 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.175799    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.177944 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.177926    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.178037 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.177990    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:10.180091 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.180075    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.182342 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.182316    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.186138 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.186115    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.188897 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.188878    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.191440 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191420    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-multus\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.191551 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-tmp-dir\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.191551 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191490    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-sys\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.191551 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191513    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-hosts-file\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.191679 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkv5\" (UniqueName: \"kubernetes.io/projected/e041a721-484d-4b8b-8611-fa6d4d8c4602-kube-api-access-9fkv5\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.191679 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191556    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"5a65f0fc8a51566f4a4d62ed9c49d84d","Type":"ContainerStarted","Data":"dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42"}
Nov 01 06:09:10.191679 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191605    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"23d718e8dfa8061bd8b70748a702b624","Type":"ContainerStarted","Data":"a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3"}
Nov 01 06:09:10.191679 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-tmp\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.191819 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.191819 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191708    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2svs\" (UniqueName: \"kubernetes.io/projected/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-kube-api-access-p2svs\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.191819 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b73c5c00-4399-4546-a2fe-7f58e086e5a8-konnectivity-ca\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.191819 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-netns\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.191819 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191784    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-kubelet-config\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b73c5c00-4399-4546-a2fe-7f58e086e5a8-agent-certs\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-conf\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-host\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-conf-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-daemon-config\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191964    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-lib-modules\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.191988    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf54\" (UniqueName: \"kubernetes.io/projected/1dd22b57-4397-461a-ba98-57179378a67d-kube-api-access-qcf54\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192058 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192045    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-hostroot\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bss\" (UniqueName: \"kubernetes.io/projected/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-kube-api-access-85bss\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192104    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-kubernetes\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192122    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cnibin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cni-binary-copy\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192150    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-dbus\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-system-cni-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192192    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-iptables-alerter-script\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-host-slash\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192221    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-tuned\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192256    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-kubelet\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192357 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192365    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysconfig\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qt59\" (UniqueName: \"kubernetes.io/projected/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-kube-api-access-6qt59\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-var-lib-kubelet\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dd22b57-4397-461a-ba98-57179378a67d-host\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-system-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-k8s-cni-cncf-io\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-bin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-multus-certs\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192613    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2cc\" (UniqueName: \"kubernetes.io/projected/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-kube-api-access-zw2cc\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-modprobe-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-etc-kubernetes\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192796    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-cnibin\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-systemd\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.192857 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-run\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.193394 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-socket-dir-parent\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.193394 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.193394 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.193394 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxsf\" (UniqueName: \"kubernetes.io/projected/2af01093-4482-4d87-8d74-6de00dde3fd4-kube-api-access-kkxsf\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.193394 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.192972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dd22b57-4397-461a-ba98-57179378a67d-serviceca\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.216983 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.216955    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.216983 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.216955    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.217103 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.217025    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m98z\""
Nov 01 06:09:10.218690 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.218673    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwtj8\""
Nov 01 06:09:10.218772 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.218761    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 06:09:10.218866 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.218851    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 06:09:10.218942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.218929    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 06:09:10.218976 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.218952    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.219130 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.219117    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.222838 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.222823    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.223251 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.223233    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.223307 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.223284    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 06:09:10.223348 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.223341    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wss87\""
Nov 01 06:09:10.225044 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.225029    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 06:09:10.225136 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.225125    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnlbn\""
Nov 01 06:09:10.227219 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.227205    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.227444 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.227432    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.227588 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.227559    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnk2x\""
Nov 01 06:09:10.232046 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.232015    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vsdrw\""
Nov 01 06:09:10.232112 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.232059    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 06:09:10.252936 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.252908    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 06:09:10.253070 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.252992    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.253070 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.253052    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 06:09:10.253070 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.253052    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 06:09:10.253843 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.253759    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 06:09:10.253843 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.253806    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.254010 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.253845    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.254010 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.253997    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 06:09:10.254187 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254139    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdp5j\""
Nov 01 06:09:10.254277 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254254    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 06:09:10.254319 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254282    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78q7p\""
Nov 01 06:09:10.254398 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254384    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.254568 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254554    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 06:09:10.254625 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254557    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnjvc\""
Nov 01 06:09:10.254778 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254765    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 06:09:10.254807 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.254770    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 06:09:10.262091 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.262074    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 06:09:10.262137 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.262110    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jtf7q"
Nov 01 06:09:10.282770 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.282749    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 06:09:10.293615 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-k8s-cni-cncf-io\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.293615 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-bin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.293615 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-multus-certs\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.293749 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293626    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.293749 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-socket-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.293749 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293655    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-k8s-cni-cncf-io\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.293749 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293676    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-registration-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.293749 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293736    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-bin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.293890 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293769    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-multus-certs\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.293890 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.293890 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2cc\" (UniqueName: \"kubernetes.io/projected/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-kube-api-access-zw2cc\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.293890 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-kubelet\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294002 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.293989    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9b8\" (UniqueName: \"kubernetes.io/projected/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kube-api-access-5j9b8\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.294103 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-modprobe-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.294147 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.294177 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-etc-kubernetes\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.294212 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294185    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-netd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294212 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294197    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-modprobe-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.294350 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294312    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-etc-kubernetes\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.294481 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294360    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.294481 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-cnibin\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.294481 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294408    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-cnibin\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.294481 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294426    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-slash\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294481 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294428    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.294481 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294451    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-netns\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-systemd\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-run\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294548    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-socket-dir-parent\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-log-socket\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294586    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-systemd\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-config\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5ts\" (UniqueName: \"kubernetes.io/projected/a529fecd-b712-470e-87e4-56fa923ad3b9-kube-api-access-jp5ts\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294685    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-run\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.294760 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294685    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-socket-dir-parent\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294791    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.294834    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxsf\" (UniqueName: \"kubernetes.io/projected/2af01093-4482-4d87-8d74-6de00dde3fd4-kube-api-access-kkxsf\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.294936    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:09:10.794913587 +0000 UTC m=+3.131028382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294952    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.294973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dd22b57-4397-461a-ba98-57179378a67d-serviceca\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295092    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-multus\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.295150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-tmp-dir\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-multus\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-var-lib-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-node-log\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-sys\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295289    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-hosts-file\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295371    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295384    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dd22b57-4397-461a-ba98-57179378a67d-serviceca\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295431    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-sys\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.295451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295437    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-hosts-file\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a529fecd-b712-470e-87e4-56fa923ad3b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295494    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-tmp-dir\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkv5\" (UniqueName: \"kubernetes.io/projected/e041a721-484d-4b8b-8611-fa6d4d8c4602-kube-api-access-9fkv5\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-tmp\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295672    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295690    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2svs\" (UniqueName: \"kubernetes.io/projected/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-kube-api-access-p2svs\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b73c5c00-4399-4546-a2fe-7f58e086e5a8-konnectivity-ca\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295791    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-netns\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.295861 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-kubelet-config\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295902    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295903    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-netns\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295915    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-ovn\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295950    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-kubelet-config\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-bin\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.295999    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-env-overrides\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296052    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-script-lib\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b73c5c00-4399-4546-a2fe-7f58e086e5a8-agent-certs\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296133    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-conf\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296154    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-host\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296193    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-conf-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-daemon-config\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-lib-modules\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296257    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-host\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296261    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf54\" (UniqueName: \"kubernetes.io/projected/1dd22b57-4397-461a-ba98-57179378a67d-kube-api-access-qcf54\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296287    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b73c5c00-4399-4546-a2fe-7f58e086e5a8-konnectivity-ca\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.296315 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296299    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296314    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-conf\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296323    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-hostroot\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296364    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-conf-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296417    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-sys-fs\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296406    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bss\" (UniqueName: \"kubernetes.io/projected/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-kube-api-access-85bss\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296569    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296572    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-lib-modules\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-kubernetes\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296617    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-hostroot\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296666    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cnibin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cni-binary-copy\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296703    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-kubernetes\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296731    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cnibin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-dbus\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-system-cni-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-iptables-alerter-script\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.297088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296832    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-daemon-config\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296901    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-system-cni-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.296991    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-dbus\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-host-slash\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297042    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297057    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-tuned\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297102    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-kubelet\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297158    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysconfig\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297171    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cni-binary-copy\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297208    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-host-slash\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt59\" (UniqueName: \"kubernetes.io/projected/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-kube-api-access-6qt59\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297273    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-kubelet\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297269    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-systemd-units\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297309    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysconfig\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-device-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.298074 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.297369    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:10.298904 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.297421    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret podName:8b62b472-60a7-442e-a38f-e7ca7b5172ec nodeName:}" failed. No retries permitted until 2026-11-01 06:09:10.797403916 +0000 UTC m=+3.133518679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret") pod "global-pull-secret-syncer-v48xf" (UID: "8b62b472-60a7-442e-a38f-e7ca7b5172ec") : object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:10.298904 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.297597    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-iptables-alerter-script\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.299402 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-var-lib-kubelet\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.299515 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299431    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dd22b57-4397-461a-ba98-57179378a67d-host\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.299515 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-systemd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.299603 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-etc-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.299603 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299558    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dd22b57-4397-461a-ba98-57179378a67d-host\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.299603 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-system-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.299603 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299489    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-var-lib-kubelet\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.299724 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.299677    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-system-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.300215 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.300190    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.301892 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.300390    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b73c5c00-4399-4546-a2fe-7f58e086e5a8-agent-certs\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.301892 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.300404    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-tmp\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.302278 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.302258    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-tuned\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.351362 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.351336    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 06:09:10.351362 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.351365    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 06:09:10.351545 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.351381    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cmrk7 for pod openshift-network-diagnostics/network-check-target-7llbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:10.351545 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.351445    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7 podName:f320aeca-d424-42de-bfbc-c0ef0eaa2bc6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:10.851427088 +0000 UTC m=+3.187541873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cmrk7" (UniqueName: "kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7") pod "network-check-target-7llbh" (UID: "f320aeca-d424-42de-bfbc-c0ef0eaa2bc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:10.352129 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.352101    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkv5\" (UniqueName: \"kubernetes.io/projected/e041a721-484d-4b8b-8611-fa6d4d8c4602-kube-api-access-9fkv5\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.352221 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.352104    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2svs\" (UniqueName: \"kubernetes.io/projected/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-kube-api-access-p2svs\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.352221 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.352210    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxsf\" (UniqueName: \"kubernetes.io/projected/2af01093-4482-4d87-8d74-6de00dde3fd4-kube-api-access-kkxsf\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.352529 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.352512    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf54\" (UniqueName: \"kubernetes.io/projected/1dd22b57-4397-461a-ba98-57179378a67d-kube-api-access-qcf54\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.353294 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.353275    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2cc\" (UniqueName: \"kubernetes.io/projected/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-kube-api-access-zw2cc\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.353420 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.353396    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bss\" (UniqueName: \"kubernetes.io/projected/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-kube-api-access-85bss\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.354087 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.354068    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt59\" (UniqueName: \"kubernetes.io/projected/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-kube-api-access-6qt59\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.399945 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.399918    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.399953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.399976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-ovn\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.399996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-bin\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-env-overrides\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400025    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400038    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-script-lib\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400033    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400054    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-ovn\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400061    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-bin\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400105 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-sys-fs\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-systemd-units\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-device-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400207    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-systemd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400208    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-systemd-units\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400210    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-sys-fs\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400247    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-systemd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400256    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-device-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-etc-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-socket-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400352    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400343    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-etc-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-registration-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-kubelet\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400419    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-kubelet\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9b8\" (UniqueName: \"kubernetes.io/projected/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kube-api-access-5j9b8\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.400573 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400399    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-registration-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-netd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400499    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-slash\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-netns\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-log-socket\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-config\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400578    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-slash\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5ts\" (UniqueName: \"kubernetes.io/projected/a529fecd-b712-470e-87e4-56fa923ad3b9-kube-api-access-jp5ts\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400599    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-log-socket\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400607    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-netns\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400621    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-netd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-var-lib-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-node-log\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a529fecd-b712-470e-87e4-56fa923ad3b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400769    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-node-log\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401192 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401747 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400736    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-var-lib-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401747 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400947    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-socket-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.401747 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.400962    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.401747 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.401109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-env-overrides\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401747 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.401283    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-config\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.401747 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.401294    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-script-lib\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.403091 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.403075    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a529fecd-b712-470e-87e4-56fa923ad3b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.457664 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.457635    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5ts\" (UniqueName: \"kubernetes.io/projected/a529fecd-b712-470e-87e4-56fa923ad3b9-kube-api-access-jp5ts\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.457796 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.457638    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9b8\" (UniqueName: \"kubernetes.io/projected/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kube-api-access-5j9b8\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.476479 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.476436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:10.486978 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:10.486953    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode041a721_484d_4b8b_8611_fa6d4d8c4602.slice/crio-8482dd6839a64f637a07273778490847df98814978099f52d5c6b9f6eb39a10c WatchSource:0}: Error finding container 8482dd6839a64f637a07273778490847df98814978099f52d5c6b9f6eb39a10c: Status 404 returned error can't find the container with id 8482dd6839a64f637a07273778490847df98814978099f52d5c6b9f6eb39a10c
Nov 01 06:09:10.494572 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.494555    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:10.497154 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.497132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:10.500070 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.499980    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:10.502180 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.502157    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:10.503596 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:10.503571    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd22b57_4397_461a_ba98_57179378a67d.slice/crio-a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994 WatchSource:0}: Error finding container a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994: Status 404 returned error can't find the container with id a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994
Nov 01 06:09:10.506602 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.505352    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:10.511684 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.511556    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:10.520855 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:10.520822    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59555cdb_7ee8_4bd1_b100_a881c36ec6e9.slice/crio-e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3 WatchSource:0}: Error finding container e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3: Status 404 returned error can't find the container with id e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3
Nov 01 06:09:10.526161 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:10.526123    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3725a48c_aeac_4da8_b9b8_8f8eb5790c00.slice/crio-34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee WatchSource:0}: Error finding container 34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee: Status 404 returned error can't find the container with id 34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee
Nov 01 06:09:10.535224 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.535203    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:10.538363 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.538340    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:10.546659 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:10.546633    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d043f5_0ea8_4af9_8ec2_f7abdbc4f50b.slice/crio-bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c WatchSource:0}: Error finding container bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c: Status 404 returned error can't find the container with id bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c
Nov 01 06:09:10.548921 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:10.548902    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda529fecd_b712_470e_87e4_56fa923ad3b9.slice/crio-c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d WatchSource:0}: Error finding container c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d: Status 404 returned error can't find the container with id c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d
Nov 01 06:09:10.803111 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.803031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:10.803111 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.803103    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:10.803340 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.803156    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:10.803340 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.803222    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret podName:8b62b472-60a7-442e-a38f-e7ca7b5172ec nodeName:}" failed. No retries permitted until 2026-11-01 06:09:11.803208455 +0000 UTC m=+4.139323230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret") pod "global-pull-secret-syncer-v48xf" (UID: "8b62b472-60a7-442e-a38f-e7ca7b5172ec") : object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:10.803340 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.803243    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:10.803340 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.803298    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:09:11.803280939 +0000 UTC m=+4.139395705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:10.903723 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:10.903685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:10.903914 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.903849    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 06:09:10.903914 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.903867    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 06:09:10.903914 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.903876    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cmrk7 for pod openshift-network-diagnostics/network-check-target-7llbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:10.904049 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:10.903932    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7 podName:f320aeca-d424-42de-bfbc-c0ef0eaa2bc6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:11.903915842 +0000 UTC m=+4.240030604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmrk7" (UniqueName: "kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7") pod "network-check-target-7llbh" (UID: "f320aeca-d424-42de-bfbc-c0ef0eaa2bc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:11.170981 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.170490    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q" event={"ID":"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b","Type":"ContainerStarted","Data":"bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c"}
Nov 01 06:09:11.172554 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.172500    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvbrt" event={"ID":"1e9a30fd-cbc2-429c-bfab-788eef7ea27a","Type":"ContainerStarted","Data":"2fcf1a29e8c552c7b9d7fd40078d9e52cf0010972152dfe2ae519d056399a497"}
Nov 01 06:09:11.174040 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.173993    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5fnn" event={"ID":"1dd22b57-4397-461a-ba98-57179378a67d","Type":"ContainerStarted","Data":"a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994"}
Nov 01 06:09:11.175341 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.175313    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8rjvr" event={"ID":"e041a721-484d-4b8b-8611-fa6d4d8c4602","Type":"ContainerStarted","Data":"8482dd6839a64f637a07273778490847df98814978099f52d5c6b9f6eb39a10c"}
Nov 01 06:09:11.176722 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.176703    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d"}
Nov 01 06:09:11.177881 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.177846    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pr9kp" event={"ID":"3725a48c-aeac-4da8-b9b8-8f8eb5790c00","Type":"ContainerStarted","Data":"34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee"}
Nov 01 06:09:11.179115 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.179087    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sdt2h" event={"ID":"59555cdb-7ee8-4bd1-b100-a881c36ec6e9","Type":"ContainerStarted","Data":"e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3"}
Nov 01 06:09:11.180650 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.180625    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerStarted","Data":"5c2bdf5ef05d7a551872f8ed3bbdad91763dd77d3b81630e3ca20fb45aeb3176"}
Nov 01 06:09:11.182042 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.182017    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w6l2n" event={"ID":"b73c5c00-4399-4546-a2fe-7f58e086e5a8","Type":"ContainerStarted","Data":"9b3a8bc2e0184c576609c024b5f793d51619a3bff6c27bd4981f7bf1abe06c6c"}
Nov 01 06:09:11.265116 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.265050    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:10 +0000 UTC" deadline="2028-06-18 12:26:15.905279328 +0000 UTC"
Nov 01 06:09:11.265116 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.265110    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14286h17m4.640172817s"
Nov 01 06:09:11.812608 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.812317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:11.812608 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.812382    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:11.812608 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.812548    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:11.812608 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.812607    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:09:13.812589077 +0000 UTC m=+6.148703845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:11.812879 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.812669    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:11.812879 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.812700    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret podName:8b62b472-60a7-442e-a38f-e7ca7b5172ec nodeName:}" failed. No retries permitted until 2026-11-01 06:09:13.812689825 +0000 UTC m=+6.148804593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret") pod "global-pull-secret-syncer-v48xf" (UID: "8b62b472-60a7-442e-a38f-e7ca7b5172ec") : object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:11.913531 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:11.913272    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:11.913938 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.913796    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 06:09:11.913938 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.913829    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 06:09:11.913938 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.913842    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cmrk7 for pod openshift-network-diagnostics/network-check-target-7llbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:11.913938 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:11.913908    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7 podName:f320aeca-d424-42de-bfbc-c0ef0eaa2bc6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:13.91388819 +0000 UTC m=+6.250002973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmrk7" (UniqueName: "kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7") pod "network-check-target-7llbh" (UID: "f320aeca-d424-42de-bfbc-c0ef0eaa2bc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:12.162690 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:12.162600    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:12.162870 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:12.162729    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:12.162870 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:12.162814    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:12.162976 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:12.162882    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:12.162976 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:12.162943    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:12.163075 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:12.163016    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:12.200976 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:12.200881    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"5a65f0fc8a51566f4a4d62ed9c49d84d","Type":"ContainerStarted","Data":"f91ae8c39704e4c64c74ce07d693c899305b7766a361617b5fa753bfd11af070"}
Nov 01 06:09:12.266007 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:12.265953    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:10 +0000 UTC" deadline="2028-06-12 18:24:21.172103913 +0000 UTC"
Nov 01 06:09:12.266007 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:12.265996    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14148h15m8.906112876s"
Nov 01 06:09:13.831433 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:13.831014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:13.831433 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:13.831093    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:13.831433 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.831227    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:13.831433 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.831286    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret podName:8b62b472-60a7-442e-a38f-e7ca7b5172ec nodeName:}" failed. No retries permitted until 2026-11-01 06:09:17.831268752 +0000 UTC m=+10.167383529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret") pod "global-pull-secret-syncer-v48xf" (UID: "8b62b472-60a7-442e-a38f-e7ca7b5172ec") : object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:13.831433 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.831355    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:13.831433 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.831388    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:09:17.831376308 +0000 UTC m=+10.167491086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:13.932189 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:13.932115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:13.932359 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.932280    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 06:09:13.932359 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.932301    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 06:09:13.932359 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.932314    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cmrk7 for pod openshift-network-diagnostics/network-check-target-7llbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:13.932541 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:13.932366    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7 podName:f320aeca-d424-42de-bfbc-c0ef0eaa2bc6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:17.932349915 +0000 UTC m=+10.268464691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmrk7" (UniqueName: "kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7") pod "network-check-target-7llbh" (UID: "f320aeca-d424-42de-bfbc-c0ef0eaa2bc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:14.165983 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:14.164515    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:14.165983 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:14.164646    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:14.165983 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:14.164711    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:14.165983 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:14.164770    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:14.165983 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:14.164822    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:14.165983 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:14.164882    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:15.222651 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:15.222570    2572 generic.go:356] "Generic (PLEG): container finished" podID="23d718e8dfa8061bd8b70748a702b624" containerID="2cc3ac5b933865f4a479b98eb090a7eab1d5b96c556cc45a7811a5a342e8fe9c" exitCode=0
Nov 01 06:09:15.223109 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:15.222675    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"23d718e8dfa8061bd8b70748a702b624","Type":"ContainerDied","Data":"2cc3ac5b933865f4a479b98eb090a7eab1d5b96c556cc45a7811a5a342e8fe9c"}
Nov 01 06:09:15.228111 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:15.227345    2572 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="6f9d2ca375d0c6da3a7cb38fa63ed90fcd97f88eadb9b91df122692f81fa6fed" exitCode=0
Nov 01 06:09:15.228111 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:15.227389    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerDied","Data":"6f9d2ca375d0c6da3a7cb38fa63ed90fcd97f88eadb9b91df122692f81fa6fed"}
Nov 01 06:09:15.333934 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:15.333876    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal" podStartSLOduration=5.333861903 podStartE2EDuration="5.333861903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:12.312745484 +0000 UTC m=+4.648860271" watchObservedRunningTime="2026-11-01 06:09:15.333861903 +0000 UTC m=+7.669976690"
Nov 01 06:09:16.165381 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.163925    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:16.165381 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:16.164070    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:16.165381 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.164144    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:16.165381 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:16.164219    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:16.165381 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.164282    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:16.165381 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:16.164350    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:16.231553 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.231512    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w6l2n" event={"ID":"b73c5c00-4399-4546-a2fe-7f58e086e5a8","Type":"ContainerStarted","Data":"e94a93b604fc6b48648a034ad503e7189f088aaa0fc0dbf081b452b15e983b44"}
Nov 01 06:09:16.234631 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.234601    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"23d718e8dfa8061bd8b70748a702b624","Type":"ContainerStarted","Data":"566fc3810bf3f902d06e9d31e17600651af18ba71aa59a6475c11adca3dd5e1c"}
Nov 01 06:09:16.242654 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.242617    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5fnn" event={"ID":"1dd22b57-4397-461a-ba98-57179378a67d","Type":"ContainerStarted","Data":"c77b0376578d69a5d8b04a35208fb9648075efdab17ca42e9bc2521901a28cff"}
Nov 01 06:09:16.343957 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.343894    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w6l2n" podStartSLOduration=3.479860283 podStartE2EDuration="8.343876515s" totalImagesPullingTime="4.864016232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:16.342763375 +0000 UTC m=+8.678878183" watchObservedRunningTime="2026-11-01 06:09:16.343876515 +0000 UTC m=+8.679991301"
Nov 01 06:09:16.453746 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.453547    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r5fnn" podStartSLOduration=2.927803977 podStartE2EDuration="8.45352605s" totalImagesPullingTime="5.525722073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:16.453322421 +0000 UTC m=+8.789437210" watchObservedRunningTime="2026-11-01 06:09:16.45352605 +0000 UTC m=+8.789640835"
Nov 01 06:09:16.565399 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:16.565336    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" podStartSLOduration=6.565316489 podStartE2EDuration="6.565316489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:16.565003442 +0000 UTC m=+8.901118231" watchObservedRunningTime="2026-11-01 06:09:16.565316489 +0000 UTC m=+8.901431275"
Nov 01 06:09:17.249400 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:17.248395    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q" event={"ID":"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b","Type":"ContainerStarted","Data":"e95db37f5ab371b33090265369b7cf5a3364cc575594b977554b883073181d81"}
Nov 01 06:09:17.866099 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:17.866054    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:17.866266 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:17.866124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:17.866266 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.866249    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:17.866392 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.866315    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:09:25.866294365 +0000 UTC m=+18.202409143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:17.866392 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.866382    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:17.866529 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.866417    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret podName:8b62b472-60a7-442e-a38f-e7ca7b5172ec nodeName:}" failed. No retries permitted until 2026-11-01 06:09:25.866405666 +0000 UTC m=+18.202520429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret") pod "global-pull-secret-syncer-v48xf" (UID: "8b62b472-60a7-442e-a38f-e7ca7b5172ec") : object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:17.966860 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:17.966817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:17.967027 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.967006    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 06:09:17.967100 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.967030    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 06:09:17.967100 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.967044    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cmrk7 for pod openshift-network-diagnostics/network-check-target-7llbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:17.967201 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:17.967101    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7 podName:f320aeca-d424-42de-bfbc-c0ef0eaa2bc6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:25.967083866 +0000 UTC m=+18.303198643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmrk7" (UniqueName: "kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7") pod "network-check-target-7llbh" (UID: "f320aeca-d424-42de-bfbc-c0ef0eaa2bc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:18.163671 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:18.163404    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:18.163671 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:18.163431    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:18.163671 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:18.163530    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:18.163671 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:18.163628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:18.163897 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:18.163701    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:18.163897 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:18.163777    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:18.669531 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:18.668896    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:18.670731 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:18.670488    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:19.102763 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.102734    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 06:09:19.146721 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.146599    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T06:09:19.102757697Z","UUID":"1469e650-30fc-415c-9f57-8dcffd473834","Handler":null,"Name":"","Endpoint":""}
Nov 01 06:09:19.148637 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.148612    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 06:09:19.148730 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.148649    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 06:09:19.254824 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.254363    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sdt2h" event={"ID":"59555cdb-7ee8-4bd1-b100-a881c36ec6e9","Type":"ContainerStarted","Data":"75ae4b0b46a9a43d8f2b9d52c639b44898a9d6681cd77d6a12cb6e6bf3906d8a"}
Nov 01 06:09:19.256739 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.256704    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q" event={"ID":"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b","Type":"ContainerStarted","Data":"18587d1531b5f665c845b0f8b413ebf03e64ec37e94dfab0820f5f61c3f45c45"}
Nov 01 06:09:19.259841 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.259655    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvbrt" event={"ID":"1e9a30fd-cbc2-429c-bfab-788eef7ea27a","Type":"ContainerStarted","Data":"99082b5c743b3e71aabd73db02371570f20702b954d4436aa9d8b660c597b6f5"}
Nov 01 06:09:19.259841 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.259700    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:19.260101 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.260080    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:19.370868 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.370807    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sdt2h" podStartSLOduration=3.131427653 podStartE2EDuration="11.370788997s" totalImagesPullingTime="8.239361344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:19.369896461 +0000 UTC m=+11.706011248" watchObservedRunningTime="2026-11-01 06:09:19.370788997 +0000 UTC m=+11.706903782"
Nov 01 06:09:19.598422 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:19.598367    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pvbrt" podStartSLOduration=3.409041151 podStartE2EDuration="11.598353236s" totalImagesPullingTime="8.189312085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:19.483228906 +0000 UTC m=+11.819343692" watchObservedRunningTime="2026-11-01 06:09:19.598353236 +0000 UTC m=+11.934468020"
Nov 01 06:09:20.164636 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:20.163896    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:20.164636 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:20.164233    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:20.164636 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:20.164314    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:20.164636 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:20.164389    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:20.164636 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:20.164446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:20.164636 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:20.164541    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:20.264020 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:20.263984    2572 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="4f8b8668a942c2509efb69e70e74114a54bd38bfa1998830e80f541797076bfe" exitCode=0
Nov 01 06:09:20.264200 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:20.264107    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerDied","Data":"4f8b8668a942c2509efb69e70e74114a54bd38bfa1998830e80f541797076bfe"}
Nov 01 06:09:21.269202 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:21.269120    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q" event={"ID":"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b","Type":"ContainerStarted","Data":"b07b354cf1d6d0e4f46906bec44fed55782ebab47ffcb96d9ed3251f5c3cf925"}
Nov 01 06:09:21.271635 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:21.271219    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pr9kp" event={"ID":"3725a48c-aeac-4da8-b9b8-8f8eb5790c00","Type":"ContainerStarted","Data":"7315436d337241b281292979aaf129c43d318401e89d0f54bd7d75a7fb7fee8e"}
Nov 01 06:09:21.386492 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:21.386416    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q" podStartSLOduration=3.750723557 podStartE2EDuration="13.386402051s" totalImagesPullingTime="9.635678494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:21.385707631 +0000 UTC m=+13.721822417" watchObservedRunningTime="2026-11-01 06:09:21.386402051 +0000 UTC m=+13.722516836"
Nov 01 06:09:21.497951 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:21.497904    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pr9kp" podStartSLOduration=4.7980007140000005 podStartE2EDuration="13.497891042s" totalImagesPullingTime="8.699890328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:21.497787466 +0000 UTC m=+13.833902252" watchObservedRunningTime="2026-11-01 06:09:21.497891042 +0000 UTC m=+13.834005842"
Nov 01 06:09:22.162106 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.162072    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:22.162280 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.162133    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:22.162280 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.162128    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:22.162280 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:22.162247    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:22.162434 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:22.162327    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:22.162434 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:22.162392    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:22.275819 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.275786    2572 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="b68e016ce9786ebc5e0e757cc6fdcca60377110b617357a745216a224d30b24a" exitCode=0
Nov 01 06:09:22.276454 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.275846    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerDied","Data":"b68e016ce9786ebc5e0e757cc6fdcca60377110b617357a745216a224d30b24a"}
Nov 01 06:09:22.277502 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.277411    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8rjvr" event={"ID":"e041a721-484d-4b8b-8611-fa6d4d8c4602","Type":"ContainerStarted","Data":"c1d165aba4fdd8ca027af015c9f122c5ecfcb0c76116106a6c17e8a27faf44a4"}
Nov 01 06:09:22.504088 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:22.503982    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8rjvr" podStartSLOduration=3.600403395 podStartE2EDuration="14.503966631s" totalImagesPullingTime="10.903563236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:22.503638956 +0000 UTC m=+14.839753752" watchObservedRunningTime="2026-11-01 06:09:22.503966631 +0000 UTC m=+14.840081415"
Nov 01 06:09:24.164126 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:24.164086    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:24.164820 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:24.164227    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:24.164820 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:24.164589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:24.164820 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:24.164590    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:24.164820 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:24.164663    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:24.164820 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:24.164786    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:24.284214 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:24.284179    2572 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="b4a1e8cc05d4651f0a22e379cff971febb5bf1fb275c814819367fa6f4a9a967" exitCode=0
Nov 01 06:09:24.284369 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:24.284227    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerDied","Data":"b4a1e8cc05d4651f0a22e379cff971febb5bf1fb275c814819367fa6f4a9a967"}
Nov 01 06:09:25.933197 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:25.932966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:25.933570 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:25.933244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:25.933570 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:25.933136    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:25.933570 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:25.933340    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:09:41.933324329 +0000 UTC m=+34.269439092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 06:09:25.933816 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:25.933799    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:25.933895 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:25.933857    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret podName:8b62b472-60a7-442e-a38f-e7ca7b5172ec nodeName:}" failed. No retries permitted until 2026-11-01 06:09:41.933841878 +0000 UTC m=+34.269956645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret") pod "global-pull-secret-syncer-v48xf" (UID: "8b62b472-60a7-442e-a38f-e7ca7b5172ec") : object "kube-system"/"original-pull-secret" not registered
Nov 01 06:09:26.033856 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.033668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:26.033993 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.033859    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 06:09:26.033993 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.033885    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 06:09:26.033993 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.033905    2572 projected.go:196] Error preparing data for projected volume kube-api-access-cmrk7 for pod openshift-network-diagnostics/network-check-target-7llbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:26.033993 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.033972    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7 podName:f320aeca-d424-42de-bfbc-c0ef0eaa2bc6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.033954655 +0000 UTC m=+34.370069425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmrk7" (UniqueName: "kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7") pod "network-check-target-7llbh" (UID: "f320aeca-d424-42de-bfbc-c0ef0eaa2bc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 06:09:26.162068 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.161999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:26.162068 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.162010    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:26.162225 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.162126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:26.162225 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.162135    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:26.162225 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.162206    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:26.162333 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:26.162254    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:26.291536 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.291500    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerStarted","Data":"7f1e2d682260caffce568fd9be95e764dd688a419dc30611b52c5a0bf9a97910"}
Nov 01 06:09:26.294412 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.294385    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"a05f300313b0ac460b7930af42b9720238763319030fbdb45926ba0eed92f93e"}
Nov 01 06:09:26.294572 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.294455    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"1d293baedf5a398b2436f074b8d23bd39c55ac1db87946ec1a5115b5145ff857"}
Nov 01 06:09:26.294572 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.294516    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"4d25275c7de4245c573e312f976a1dc6cc44489a6dca385df1203eb57b1d643b"}
Nov 01 06:09:26.294572 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.294545    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"d229a2b093a439d378f8c562ff5fe6f6728fb3e5a80e85d28c6bb2e50a98f626"}
Nov 01 06:09:26.294725 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.294579    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"a7f3d01dfa69d28d935116bb89bf92d3d12235c0e9583cc230b5d1527dbe55fd"}
Nov 01 06:09:26.294725 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:26.294606    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"73e2a59c41e1d7c3fa24c6b7f3899c38f19669c3778f7d67d1ebbb7bf49c4b70"}
Nov 01 06:09:27.298801 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:27.298769    2572 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="7f1e2d682260caffce568fd9be95e764dd688a419dc30611b52c5a0bf9a97910" exitCode=0
Nov 01 06:09:27.299171 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:27.298816    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerDied","Data":"7f1e2d682260caffce568fd9be95e764dd688a419dc30611b52c5a0bf9a97910"}
Nov 01 06:09:28.162974 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:28.162936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:28.163145 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:28.163017    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:28.163145 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:28.163113    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:28.163247 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:28.163204    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:28.163301 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:28.163249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:28.163351 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:28.163300    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:28.304111 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:28.304080    2572 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="3be5fcc3b7eb6b6d3e2562fd8f93ee619aa615fbf99b9806f4aae430ab2063af" exitCode=0
Nov 01 06:09:28.304623 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:28.304162    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerDied","Data":"3be5fcc3b7eb6b6d3e2562fd8f93ee619aa615fbf99b9806f4aae430ab2063af"}
Nov 01 06:09:28.307493 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:28.307453    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"8c3fdff2d65511f3024e2a5d3b00ed3f72442727bb8fb81bb84bdeb0d1f92f37"}
Nov 01 06:09:29.312549 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:29.312513    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bq58b" event={"ID":"2af01093-4482-4d87-8d74-6de00dde3fd4","Type":"ContainerStarted","Data":"c7b96198019b55e73deb58007d6c1f961c3fb3cadf00d23576ec0856ff705a92"}
Nov 01 06:09:29.529565 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:29.529516    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bq58b" podStartSLOduration=9.127779072 podStartE2EDuration="21.529502837s" totalImagesPullingTime="12.401723765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:29.529345528 +0000 UTC m=+21.865460312" watchObservedRunningTime="2026-11-01 06:09:29.529502837 +0000 UTC m=+21.865617622"
Nov 01 06:09:30.161673 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:30.161639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:30.161905 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:30.161636    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:30.161905 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:30.161650    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:30.161905 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:30.161866    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:30.162064 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:30.161756    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:30.162064 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:30.161924    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:30.317992 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:30.317915    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" event={"ID":"a529fecd-b712-470e-87e4-56fa923ad3b9","Type":"ContainerStarted","Data":"1e2b39a82d67c9afff42454389e588e247bfdb52fb80b0169cf459274f4124f6"}
Nov 01 06:09:30.495824 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:30.495787    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:30.541323 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:30.541269    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc" podStartSLOduration=7.505584604 podStartE2EDuration="22.541256775s" totalImagesPullingTime="15.035672171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:30.541194687 +0000 UTC m=+22.877309471" watchObservedRunningTime="2026-11-01 06:09:30.541256775 +0000 UTC m=+22.877371560"
Nov 01 06:09:31.321439 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:31.321295    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:31.321439 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:31.321387    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:31.339112 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:31.338894    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:31.339436 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:31.339421    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:32.162004 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.161973    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:32.162004 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.161994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:32.162185 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:32.162089    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:32.162185 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.162130    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:32.162250 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:32.162199    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:32.162283 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:32.162264    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:32.841647 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.841616    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v48xf"]
Nov 01 06:09:32.842127 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.841724    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:32.842127 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:32.841823    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:32.844376 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.844343    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7llbh"]
Nov 01 06:09:32.844516 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.844420    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:32.844589 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:32.844521    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:32.844957 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.844934    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6492"]
Nov 01 06:09:32.845177 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:32.845154    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:32.845300 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:32.845280    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:34.161922 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:34.161890    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:34.162288 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:34.161928    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:34.162288 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:34.162041    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:34.162288 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:34.162150    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:35.161694 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:35.161616    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:35.161856 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:35.161731    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:36.161876 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:36.161842    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:36.162221 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:36.161842    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:36.162221 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:36.161961    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6492" podUID="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae"
Nov 01 06:09:36.162221 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:36.162042    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7llbh" podUID="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6"
Nov 01 06:09:37.161679 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:37.161647    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:37.161851 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:37.161757    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v48xf" podUID="8b62b472-60a7-442e-a38f-e7ca7b5172ec"
Nov 01 06:09:37.896830 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:37.896800    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeReady"
Nov 01 06:09:37.897200 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:37.896910    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 06:09:38.007829 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.007782    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-whgfv"]
Nov 01 06:09:38.011016 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.010909    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-99jhh"]
Nov 01 06:09:38.011168 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.011093    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.013756 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.013736    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.063341 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.063310    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 06:09:38.063341 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.063313    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8pggw\""
Nov 01 06:09:38.063341 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.063320    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 06:09:38.063568 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.063320    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 06:09:38.079514 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.079490    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whgfv"]
Nov 01 06:09:38.082193 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.082160    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-99jhh"]
Nov 01 06:09:38.120970 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.120937    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c5066e-78ec-49ff-ba9f-bec5967833b9-tmp-dir\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.120970 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.120970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6hcn\" (UniqueName: \"kubernetes.io/projected/82c5066e-78ec-49ff-ba9f-bec5967833b9-kube-api-access-j6hcn\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.121169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.121011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82c5066e-78ec-49ff-ba9f-bec5967833b9-config-volume\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.121169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.121077    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.121169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.121137    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsvh\" (UniqueName: \"kubernetes.io/projected/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-kube-api-access-4vsvh\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.121169 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.121164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-cert\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.161849 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.161793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:38.161849 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.161811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:38.163535 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.163509    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 06:09:38.163640 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.163574    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgqp9\""
Nov 01 06:09:38.163754 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.163743    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 06:09:38.214163 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.214132    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 06:09:38.214163 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.214141    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 06:09:38.214163 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.214137    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 06:09:38.214163 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.214159    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8r6tp\""
Nov 01 06:09:38.214406 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.214176    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h2lht\""
Nov 01 06:09:38.221584 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.221548    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsvh\" (UniqueName: \"kubernetes.io/projected/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-kube-api-access-4vsvh\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.221682 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.221594    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-cert\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.221835 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.221817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c5066e-78ec-49ff-ba9f-bec5967833b9-tmp-dir\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.221883 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.221851    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hcn\" (UniqueName: \"kubernetes.io/projected/82c5066e-78ec-49ff-ba9f-bec5967833b9-kube-api-access-j6hcn\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.221932 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.221911    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82c5066e-78ec-49ff-ba9f-bec5967833b9-config-volume\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.221973 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.221934    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.222052 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:38.222037    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:38.222101 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:38.222097    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:38.722081522 +0000 UTC m=+31.058196300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:38.222361 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.222344    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c5066e-78ec-49ff-ba9f-bec5967833b9-tmp-dir\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.222489 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.222450    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82c5066e-78ec-49ff-ba9f-bec5967833b9-config-volume\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.225918 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.225804    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-cert\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.281213 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.281188    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6hcn\" (UniqueName: \"kubernetes.io/projected/82c5066e-78ec-49ff-ba9f-bec5967833b9-kube-api-access-j6hcn\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.281448 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.281425    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsvh\" (UniqueName: \"kubernetes.io/projected/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-kube-api-access-4vsvh\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.323770 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.323734    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:38.701029 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.700996    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whgfv"]
Nov 01 06:09:38.704452 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:38.704423    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adfb865_b3de_4b4e_b9c7_ae82f11456d6.slice/crio-6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6 WatchSource:0}: Error finding container 6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6: Status 404 returned error can't find the container with id 6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6
Nov 01 06:09:38.726319 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.726290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:38.726461 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:38.726439    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:38.726548 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:38.726537    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:39.726518719 +0000 UTC m=+32.062633499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:38.910599 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.910560    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q8k87"]
Nov 01 06:09:38.918190 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.918164    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:38.971353 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.971276    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 06:09:38.971619 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.971587    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 06:09:38.971619 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.971612    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 06:09:38.971864 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.971617    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 06:09:38.971864 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.971686    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q8k87"]
Nov 01 06:09:38.971864 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:38.971715    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wr\""
Nov 01 06:09:39.028532 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.028497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-crio-socket\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.028713 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.028539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.028775 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.028703    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczgk\" (UniqueName: \"kubernetes.io/projected/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-api-access-tczgk\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.028775 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.028743    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-data-volume\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.028884 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.028830    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.129955 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.129915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.130119 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.129973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-crio-socket\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.130119 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.130087    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:39.130246 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.130113    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.130246 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.130145    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-crio-socket\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.130246 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.130155    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:39.630134445 +0000 UTC m=+31.966249217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:39.130246 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.130178    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 06:09:39.130436 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.130243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczgk\" (UniqueName: \"kubernetes.io/projected/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-api-access-tczgk\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.130436 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.130276    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-data-volume\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.130436 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.130322    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:39.630302944 +0000 UTC m=+31.966417711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : configmap "kube-rbac-proxy" not found
Nov 01 06:09:39.130661 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.130644    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-data-volume\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.161583 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.161559    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:39.162510 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.162488    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dn84m-krqpd"]
Nov 01 06:09:39.189326 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.189301    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczgk\" (UniqueName: \"kubernetes.io/projected/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-api-access-tczgk\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.208942 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.208920    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.213726 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.213706    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 06:09:39.213863 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.213769    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rlspf\""
Nov 01 06:09:39.221925 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.221879    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dn84m-krqpd"]
Nov 01 06:09:39.260963 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.260933    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 06:09:39.312296 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.312251    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 06:09:39.313686 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.313670    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jp6zc\""
Nov 01 06:09:39.331309 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.331281    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.331401 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.331334    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.331451 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.331407    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.331515 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.331502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.339927 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.339904    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whgfv" event={"ID":"7adfb865-b3de-4b4e-b9c7-ae82f11456d6","Type":"ContainerStarted","Data":"6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6"}
Nov 01 06:09:39.432740 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.432712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.432905 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.432774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.432905 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.432799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.432905 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.432826    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.433218 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.433193    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.433437 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.433421    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.433610 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.433581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.489032 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.488956    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.536111 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.536077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:39.634106 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.634070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.634269 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.634207    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 06:09:39.634269 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.634238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:39.634376 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.634271    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.63425757 +0000 UTC m=+32.970372364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : configmap "kube-rbac-proxy" not found
Nov 01 06:09:39.634414 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.634382    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:39.634446 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.634440    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.634421982 +0000 UTC m=+32.970536747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:39.735378 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.735345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:39.735606 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.735543    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:39.735651 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:39.735622    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:41.73560649 +0000 UTC m=+34.071721256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:39.917201 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:39.917160    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dn84m-krqpd"]
Nov 01 06:09:39.921563 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:39.921525    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf40282_bfe4_4279_8304_3173ec43dcf1.slice/crio-d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1 WatchSource:0}: Error finding container d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1: Status 404 returned error can't find the container with id d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1
Nov 01 06:09:40.343274 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.343236    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dn84m-krqpd" event={"ID":"cbf40282-bfe4-4279-8304-3173ec43dcf1","Type":"ContainerStarted","Data":"d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1"}
Nov 01 06:09:40.547344 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.547307    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"]
Nov 01 06:09:40.579651 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.579613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:40.607963 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.607900    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"]
Nov 01 06:09:40.641749 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.641718    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:40.641919 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.641798    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:40.641919 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:40.641846    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 06:09:40.641919 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:40.641905    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:40.642052 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:40.641922    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.641902101 +0000 UTC m=+34.978016871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : configmap "kube-rbac-proxy" not found
Nov 01 06:09:40.642052 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:40.641954    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.641939509 +0000 UTC m=+34.978054275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:40.707364 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.707337    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9gkn9\""
Nov 01 06:09:40.708198 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.708172    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 06:09:40.708288 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.708230    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 06:09:40.743076 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.743044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wpf\" (UniqueName: \"kubernetes.io/projected/c8dc270b-a9e6-46cc-b99c-9452eec6e013-kube-api-access-g6wpf\") pod \"migrator-66df5d496b-5n8jp\" (UID: \"c8dc270b-a9e6-46cc-b99c-9452eec6e013\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:40.844150 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.844112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wpf\" (UniqueName: \"kubernetes.io/projected/c8dc270b-a9e6-46cc-b99c-9452eec6e013-kube-api-access-g6wpf\") pod \"migrator-66df5d496b-5n8jp\" (UID: \"c8dc270b-a9e6-46cc-b99c-9452eec6e013\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:40.904496 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.904395    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wpf\" (UniqueName: \"kubernetes.io/projected/c8dc270b-a9e6-46cc-b99c-9452eec6e013-kube-api-access-g6wpf\") pod \"migrator-66df5d496b-5n8jp\" (UID: \"c8dc270b-a9e6-46cc-b99c-9452eec6e013\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:40.906164 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:40.906139    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:41.304052 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:41.304022    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"]
Nov 01 06:09:41.313598 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:41.313572    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dc270b_a9e6_46cc_b99c_9452eec6e013.slice/crio-d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d WatchSource:0}: Error finding container d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d: Status 404 returned error can't find the container with id d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d
Nov 01 06:09:41.347507 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:41.347453    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp" event={"ID":"c8dc270b-a9e6-46cc-b99c-9452eec6e013","Type":"ContainerStarted","Data":"d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d"}
Nov 01 06:09:41.753514 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:41.753409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:41.753658 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:41.753577    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:41.753658 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:41.753644    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:45.753628288 +0000 UTC m=+38.089743054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:41.954810 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:41.954769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:41.954966 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:41.954868    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:41.954966 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:41.954933    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 06:09:41.955044 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:41.955004    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs podName:c618c3d8-5fa0-4e2e-a222-81b672c9b0ae nodeName:}" failed. No retries permitted until 2026-11-01 06:10:13.954987249 +0000 UTC m=+66.291102032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs") pod "network-metrics-daemon-l6492" (UID: "c618c3d8-5fa0-4e2e-a222-81b672c9b0ae") : secret "metrics-daemon-secret" not found
Nov 01 06:09:41.957302 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:41.957282    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:42.055752 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.055718    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:42.058417 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.058395    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:42.073525 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.073502    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:42.218965 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.218933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:42.352739 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.352664    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whgfv" event={"ID":"7adfb865-b3de-4b4e-b9c7-ae82f11456d6","Type":"ContainerStarted","Data":"ad8e9a4b8dbaf1bc329830a3de549c939dee5b7ea4572a4ab2b6fd4b59795906"}
Nov 01 06:09:42.471284 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.470925    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-whgfv" podStartSLOduration=2.114262547 podStartE2EDuration="5.470905737s" totalImagesPullingTime="3.35664319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:42.46963765 +0000 UTC m=+34.805752435" watchObservedRunningTime="2026-11-01 06:09:42.470905737 +0000 UTC m=+34.807020523"
Nov 01 06:09:42.520602 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.520570    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7llbh"]
Nov 01 06:09:42.524742 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:42.524713    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf320aeca_d424_42de_bfbc_c0ef0eaa2bc6.slice/crio-fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d WatchSource:0}: Error finding container fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d: Status 404 returned error can't find the container with id fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d
Nov 01 06:09:42.619980 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.619848    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v48xf"]
Nov 01 06:09:42.624656 ip-10-0-138-21 kubenswrapper[2572]: W1101 06:09:42.624616    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b62b472_60a7_442e_a38f_e7ca7b5172ec.slice/crio-c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb WatchSource:0}: Error finding container c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb: Status 404 returned error can't find the container with id c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb
Nov 01 06:09:42.662506 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.662431    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:42.662638 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:42.662552    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:42.662638 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:42.662581    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:42.662714 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:42.662642    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.662625838 +0000 UTC m=+38.998740613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:42.662714 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:42.662665    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 06:09:42.662825 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:42.662721    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.662709654 +0000 UTC m=+38.998824440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : configmap "kube-rbac-proxy" not found
Nov 01 06:09:43.355756 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:43.355716    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7llbh" event={"ID":"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6","Type":"ContainerStarted","Data":"fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d"}
Nov 01 06:09:43.357198 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:43.357152    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dn84m-krqpd" event={"ID":"cbf40282-bfe4-4279-8304-3173ec43dcf1","Type":"ContainerStarted","Data":"f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069"}
Nov 01 06:09:43.358283 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:43.358265    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v48xf" event={"ID":"8b62b472-60a7-442e-a38f-e7ca7b5172ec","Type":"ContainerStarted","Data":"c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb"}
Nov 01 06:09:43.472794 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:43.472744    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dn84m-krqpd" podStartSLOduration=1.289015636 podStartE2EDuration="4.472729163s" totalImagesPullingTime="3.183713527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:43.469397027 +0000 UTC m=+35.805511812" watchObservedRunningTime="2026-11-01 06:09:43.472729163 +0000 UTC m=+35.808843947"
Nov 01 06:09:44.363527 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:44.363488    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp" event={"ID":"c8dc270b-a9e6-46cc-b99c-9452eec6e013","Type":"ContainerStarted","Data":"29bc0b0ab4578d2ee426ef9c1e160baad9e7142b0f78bd50f70e7c5652a82466"}
Nov 01 06:09:44.476969 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:44.476906    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp" podStartSLOduration=2.376514725 podStartE2EDuration="4.476887031s" totalImagesPullingTime="2.100372306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:44.475977642 +0000 UTC m=+36.812092427" watchObservedRunningTime="2026-11-01 06:09:44.476887031 +0000 UTC m=+36.813001817"
Nov 01 06:09:45.787308 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:45.787234    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:45.787751 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:45.787429    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:45.787751 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:45.787532    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.787511001 +0000 UTC m=+46.123625783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:46.695736 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:46.695693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:46.695940 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:46.695790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:46.695940 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:46.695836    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 06:09:46.695940 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:46.695897    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:46.695940 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:46.695919    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:54.695897606 +0000 UTC m=+47.032012384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : configmap "kube-rbac-proxy" not found
Nov 01 06:09:46.696129 ip-10-0-138-21 kubenswrapper[2572]: E1101 06:09:46.695944    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:54.695927402 +0000 UTC m=+47.032042180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:47.026091 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:47.026024    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sdt2h_59555cdb-7ee8-4bd1-b100-a881c36ec6e9/dns-node-resolver/0.log"
Nov 01 06:09:47.372784 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:47.372742    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7llbh" event={"ID":"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6","Type":"ContainerStarted","Data":"25e83cca079b7425d3bab22d0c82c8f837decc8a2b1996acde6449f14f075e94"}
Nov 01 06:09:47.373001 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:47.372967    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:48.027314 ip-10-0-138-21 kubenswrapper[2572]: I1101 06:09:48.027286    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5fnn_1dd22b57-4397-461a-ba98-57179378a67d/node-ca/0.log"
Nov 01 06:09:48.359310 ip-10-0-138-21 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 06:09:48.376941 ip-10-0-138-21 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 06:09:48.377118 ip-10-0-138-21 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 06:09:48.377191 ip-10-0-138-21 systemd[1]: kubelet.service: Consumed 2.491s CPU time.
Nov 01 06:09:48.401133 ip-10-0-138-21 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 06:09:48.499372 ip-10-0-138-21 kubenswrapper[4468]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 06:09:48.499372 ip-10-0-138-21 kubenswrapper[4468]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 06:09:48.499372 ip-10-0-138-21 kubenswrapper[4468]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 06:09:48.499372 ip-10-0-138-21 kubenswrapper[4468]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 06:09:48.501426 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501339    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.501426 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501426    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501431    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501434    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501437    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501440    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501444    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501446    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501449    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501452    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501455    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501458    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501460    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501477    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501480    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501489    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501492    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501494    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501498    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501501    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.501508 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501503    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501506    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501508    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501511    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501514    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501517    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501520    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501523    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501526    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501528    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501531    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501534    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501536    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501539    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501541    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501546    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501550    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501553    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501556    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.501983 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501558    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501561    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501563    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501565    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501568    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501571    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501573    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501576    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501580    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501582    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501585    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501587    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501590    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501593    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501596    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501598    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501601    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501604    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501607    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501609    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.502460 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501612    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501614    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501617    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501619    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501622    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501624    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501627    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501630    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501632    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501635    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501637    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501640    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501642    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501645    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501647    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501650    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501652    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501655    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501658    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.502976 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501660    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501663    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501666    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501669    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501671    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501674    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501676    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501679    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501681    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501684    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501686    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501688    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501691    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501693    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501696    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501698    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501701    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501703    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501706    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501708    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.503450 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501711    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501713    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501719    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501722    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501726    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501729    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501731    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.501735    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502575    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502583    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502586    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502590    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502593    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502596    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502598    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502607    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502610    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502612    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502615    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502617    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.503949 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502620    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502622    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502625    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502628    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502630    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502633    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502636    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502638    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502641    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502643    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502647    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502650    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502653    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502656    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502658    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502661    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502663    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502666    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502668    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.504493 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502671    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502673    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502676    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502678    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502681    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502684    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502687    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502689    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502692    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502702    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502705    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502707    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502710    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502712    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502715    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502717    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502720    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502723    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502725    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502728    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.504961 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502730    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502733    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502736    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502738    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502741    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502743    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502748    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502758    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502762    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502765    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502767    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502770    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502772    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502775    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502777    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502781    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502785    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502788    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.505447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502791    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502793    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502796    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502798    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502806    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502810    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502812    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502815    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502817    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502820    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502822    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502825    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502828    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502830    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502833    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502835    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502838    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502840    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502842    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.505914 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502845    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502847    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502850    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502852    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502855    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502857    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502860    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502863    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502866    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502871    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502874    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502877    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502879    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502882    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502884    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502887    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502890    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.502893    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503501    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503508    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.506387 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503511    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503514    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503517    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503519    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503522    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503524    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503528    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503530    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503534    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503537    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503539    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503542    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503545    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503548    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503550    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503553    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503556    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503558    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503561    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.506903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503564    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503567    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503569    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503572    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503574    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503577    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503580    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503582    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503585    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503587    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503590    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503593    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503595    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503598    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503600    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503603    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503605    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503608    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503610    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.507364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503613    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503615    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503618    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503620    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503622    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503625    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503627    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503630    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503633    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503635    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503638    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503640    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503643    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503645    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503648    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503651    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503654    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503656    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503659    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503661    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.507837 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503664    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503666    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503669    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503671    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503674    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503677    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503680    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503682    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503685    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503687    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503690    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503692    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503695    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503698    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503700    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503703    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503705    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503708    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503710    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.508326 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503713    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503715    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503717    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503720    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503722    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503727    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503730    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503732    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503736    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503739    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503743    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503747    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503750    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503753    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503756    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503759    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503761    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503764    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503768    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503771    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.508798 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503775    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503777    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503780    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503783    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503785    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503788    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.503791    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503887    4468 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503904    4468 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503913    4468 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503920    4468 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503924    4468 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503928    4468 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503932    4468 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503941    4468 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503944    4468 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503947    4468 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503953    4468 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503956    4468 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503960    4468 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503963    4468 flags.go:64] FLAG: --cgroup-root=""
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503966    4468 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503969    4468 flags.go:64] FLAG: --client-ca-file=""
Nov 01 06:09:48.509275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503973    4468 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503976    4468 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503982    4468 flags.go:64] FLAG: --cluster-domain=""
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503984    4468 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503988    4468 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503991    4468 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503994    4468 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.503998    4468 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504002    4468 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504005    4468 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504008    4468 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504012    4468 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504014    4468 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504017    4468 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504020    4468 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504023    4468 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504028    4468 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504031    4468 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504035    4468 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504038    4468 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504041    4468 flags.go:64] FLAG: --enable-server="true"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504044    4468 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504048    4468 flags.go:64] FLAG: --event-burst="100"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504053    4468 flags.go:64] FLAG: --event-qps="50"
Nov 01 06:09:48.509905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504056    4468 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504059    4468 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504065    4468 flags.go:64] FLAG: --eviction-hard=""
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504069    4468 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504073    4468 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504076    4468 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504079    4468 flags.go:64] FLAG: --eviction-soft=""
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504082    4468 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504085    4468 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504088    4468 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504090    4468 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504093    4468 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504096    4468 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504099    4468 flags.go:64] FLAG: --feature-gates=""
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504103    4468 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504106    4468 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504110    4468 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504113    4468 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504117    4468 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504120    4468 flags.go:64] FLAG: --help="false"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504123    4468 flags.go:64] FLAG: --hostname-override="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504126    4468 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504129    4468 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 06:09:48.510564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504132    4468 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504136    4468 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504139    4468 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504142    4468 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504145    4468 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504147    4468 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504151    4468 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504154    4468 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504159    4468 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504161    4468 flags.go:64] FLAG: --kube-reserved=""
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504164    4468 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504168    4468 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504172    4468 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504175    4468 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504178    4468 flags.go:64] FLAG: --lock-file=""
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504181    4468 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504184    4468 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504187    4468 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504192    4468 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504195    4468 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504198    4468 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504201    4468 flags.go:64] FLAG: --logging-format="text"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504204    4468 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504208    4468 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 06:09:48.511761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504211    4468 flags.go:64] FLAG: --manifest-url=""
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504213    4468 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504218    4468 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504221    4468 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504225    4468 flags.go:64] FLAG: --max-pods="110"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504229    4468 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504232    4468 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504234    4468 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504237    4468 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504240    4468 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504243    4468 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504246    4468 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504253    4468 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504256    4468 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504259    4468 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504262    4468 flags.go:64] FLAG: --pod-cidr=""
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504267    4468 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504270    4468 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504273    4468 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504277    4468 flags.go:64] FLAG: --port="10250"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504281    4468 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504284    4468 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-066853833f9932c8e"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504287    4468 flags.go:64] FLAG: --qos-reserved=""
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504290    4468 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504293    4468 flags.go:64] FLAG: --register-node="true"
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504295    4468 flags.go:64] FLAG: --register-with-taints=""
Nov 01 06:09:48.512559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504300    4468 flags.go:64] FLAG: --registry-burst="10"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504303    4468 flags.go:64] FLAG: --registry-qps="5"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504305    4468 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504308    4468 flags.go:64] FLAG: --reserved-memory=""
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504312    4468 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504316    4468 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504318    4468 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504321    4468 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504324    4468 flags.go:64] FLAG: --runonce="false"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504327    4468 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504330    4468 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504334    4468 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504337    4468 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504340    4468 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504343    4468 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504346    4468 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504349    4468 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504352    4468 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504355    4468 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504358    4468 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504361    4468 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504364    4468 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504368    4468 flags.go:64] FLAG: --system-cgroups=""
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504371    4468 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504376    4468 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 06:09:48.513279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504380    4468 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504383    4468 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504388    4468 flags.go:64] FLAG: --tls-min-version=""
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504391    4468 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504394    4468 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504397    4468 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504399    4468 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504402    4468 flags.go:64] FLAG: --v="2"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504407    4468 flags.go:64] FLAG: --version="false"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504411    4468 flags.go:64] FLAG: --vmodule=""
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504415    4468 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504418    4468 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504575    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504579    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504582    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504585    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504588    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504590    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504593    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504596    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504599    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504601    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504604    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.514067 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504606    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504609    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504612    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504614    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504617    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504619    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504624    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504626    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504629    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504634    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504636    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504639    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504642    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504644    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504647    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504649    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504652    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504654    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504657    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504660    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.514678 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504662    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504665    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504667    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504670    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504673    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504675    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504678    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504680    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504683    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504686    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504690    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504694    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504697    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504699    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504702    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504704    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504707    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504709    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504713    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.515485 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504716    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504719    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504722    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504725    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504727    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504730    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504732    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504735    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504738    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504740    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504743    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504745    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504748    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504750    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504753    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504755    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504758    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504760    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504763    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.516074 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504765    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504768    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504770    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504773    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504776    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504778    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504781    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504784    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504786    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504788    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504791    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504794    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504798    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504800    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504804    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504809    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504812    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504815    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504818    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.516590 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504820    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504823    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504825    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504828    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504830    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504833    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504836    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504838    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504841    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504843    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504846    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504848    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504851    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504853    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504855    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504858    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504860    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.517089 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504863    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.504890    4468 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504926    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504930    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504933    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504936    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504939    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504941    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504945    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504948    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504951    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504953    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504957    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504959    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504962    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504965    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.517579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504968    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504970    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504973    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504976    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504979    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504982    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504986    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504989    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504992    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504995    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.504998    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505000    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505003    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505005    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505008    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505011    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505014    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505016    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.518015 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505019    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505022    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505024    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505027    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505030    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505032    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505036    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505039    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505041    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505044    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505047    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505050    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505052    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505055    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505057    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505060    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505066    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505068    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505071    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505073    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.518733 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505076    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505078    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505081    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505083    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505086    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505088    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505091    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505093    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505096    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505100    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505104    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505107    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505110    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505113    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505116    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505118    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505121    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505123    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505127    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.519278 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505129    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505133    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505135    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505138    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505140    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505143    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505145    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505148    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505150    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505153    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505155    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505158    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505160    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505163    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505165    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505167    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505170    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505172    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505175    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505177    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.519776 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505179    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505182    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505184    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505187    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505189    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505192    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505194    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505197    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505200    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505202    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505205    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505208    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505211    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505214    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.505216    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.509049    4468 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.509066    4468 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.509084    4468 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.509089    4468 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509147    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.520258 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509152    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509156    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509159    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509162    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509165    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509167    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509170    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509173    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509175    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509178    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509181    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509183    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509186    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509189    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509191    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509194    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509197    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509200    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509202    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.520815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509205    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509207    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509210    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509212    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509215    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509218    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509221    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509223    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509226    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509229    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509234    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509238    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509241    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509243    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509246    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509249    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509251    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509254    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509256    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.521335 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509258    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509262    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509265    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509267    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509270    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509272    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509275    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509278    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509281    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509284    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509287    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509290    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509293    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509295    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509298    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509300    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509303    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509305    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509308    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509311    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.521877 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509313    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509315    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509318    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509321    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509324    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509327    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509329    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509332    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509334    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509337    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509339    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509342    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509344    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509346    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509349    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509351    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509354    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509356    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509359    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509361    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.522364 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509364    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509366    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509369    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509371    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509375    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509377    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509380    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509382    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509385    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509388    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509390    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509393    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509396    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509399    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509401    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509404    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509407    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509409    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509412    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509414    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.522915 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509417    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509419    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509423    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509427    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509430    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509432    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509435    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.509481    4468 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509621    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509625    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509627    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509632    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509636    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509639    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509642    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.523586 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509645    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509648    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509650    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509653    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509655    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509658    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509660    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509663    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509665    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509668    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509670    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509673    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509675    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509678    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509682    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509684    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509687    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509689    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509692    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.524016 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509694    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509696    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509699    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509701    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509704    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509706    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509708    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509711    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509714    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509716    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509718    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509721    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509723    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509726    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509729    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509731    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509734    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509736    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509739    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509741    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.524615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509744    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509746    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509748    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509751    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509753    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509756    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509758    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509761    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509765    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509769    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509772    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509775    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509778    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509781    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509784    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509787    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509789    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509792    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509794    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.525253 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509797    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509799    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509802    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509804    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509806    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509809    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509811    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509814    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509817    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509820    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509823    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509825    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509828    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509830    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509832    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509835    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509837    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509840    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509842    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509844    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.525882 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509847    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509849    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509852    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509855    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509857    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509859    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509862    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509864    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509867    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509869    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509872    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509874    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509877    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509879    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509882    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509884    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509886    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509889    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509892    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509895    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.526389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509897    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.509919    4468 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509968    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509973    4468 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509976    4468 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509978    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509981    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509986    4468 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509990    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509993    4468 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509996    4468 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.509998    4468 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510001    4468 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510003    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510006    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 06:09:48.526931 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510009    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510011    4468 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510014    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510016    4468 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510019    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510021    4468 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510023    4468 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510026    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510028    4468 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510031    4468 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510033    4468 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510036    4468 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510039    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510041    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510047    4468 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510049    4468 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510052    4468 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510054    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510057    4468 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510059    4468 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 06:09:48.527429 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510062    4468 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510064    4468 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510067    4468 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510069    4468 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510072    4468 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510075    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510077    4468 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510080    4468 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510082    4468 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510085    4468 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510087    4468 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510091    4468 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510093    4468 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510096    4468 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510099    4468 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510101    4468 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510104    4468 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510106    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510109    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510111    4468 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 06:09:48.528272 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510114    4468 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510116    4468 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510119    4468 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510121    4468 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510124    4468 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510126    4468 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510130    4468 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510135    4468 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510138    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510141    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510144    4468 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510147    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510149    4468 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510152    4468 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510154    4468 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510157    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510159    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510162    4468 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510164    4468 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 06:09:48.529004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510167    4468 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510169    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510172    4468 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510174    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510177    4468 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510179    4468 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510182    4468 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510185    4468 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510187    4468 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510190    4468 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510192    4468 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510195    4468 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510197    4468 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510200    4468 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510202    4468 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510205    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510207    4468 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510210    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510212    4468 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 06:09:48.529519 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510217    4468 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510220    4468 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510222    4468 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510225    4468 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510227    4468 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510230    4468 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510232    4468 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510235    4468 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510237    4468 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510240    4468 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510242    4468 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510245    4468 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510247    4468 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510250    4468 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:48.510252    4468 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.510306    4468 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513003    4468 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513063    4468 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513269    4468 server.go:1014] "Starting client certificate rotation"
Nov 01 06:09:48.530004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513408    4468 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513453    4468 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 06:04:08 +0000 UTC" deadline="2028-05-02 09:33:33.958222062 +0000 UTC"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513503    4468 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13155h23m45.44472146s"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.513716    4468 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.514889    4468 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.516498    4468 log.go:25] "Validated CRI v1 runtime API"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.517889    4468 log.go:25] "Validated CRI v1 image API"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.518661    4468 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 06:09:48.530557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.520183    4468 fs.go:135] Filesystem UUIDs: map[42e24511-0162-4549-8f31-a532e06c9607:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 94f79eab-d256-4b5d-92a0-0b07450f5f63:/dev/nvme0n1p4]
Nov 01 06:09:48.530782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.520203    4468 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fcf1a29e8c552c7b9d7fd40078d9e52cf0010972152dfe2ae519d056399a497/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fcf1a29e8c552c7b9d7fd40078d9e52cf0010972152dfe2ae519d056399a497/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c2bdf5ef05d7a551872f8ed3bbdad91763dd77d3b81630e3ca20fb45aeb3176/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c2bdf5ef05d7a551872f8ed3bbdad91763dd77d3b81630e3ca20fb45aeb3176/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b3a8bc2e0184c576609c024b5f793d51619a3bff6c27bd4981f7bf1abe06c6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b3a8bc2e0184c576609c024b5f793d51619a3bff6c27bd4981f7bf1abe06c6c/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dd22b57-4397-461a-ba98-57179378a67d/volumes/kubernetes.io~projected/kube-api-access-qcf54:{mountpoint:/var/lib/kubelet/pods/1dd22b57-4397-461a-ba98-57179378a67d/volumes/kubernetes.io~projected/kube-api-access-qcf54 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e9a30fd-cbc2-429c-bfab-788eef7ea27a/volumes/kubernetes.io~projected/kube-api-access-6qt59:{mountpoint:/var/lib/kubelet/pods/1e9a30fd-cbc2-429c-bfab-788eef7ea27a/volumes/kubernetes.io~projected/kube-api-access-6qt59 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2af01093-4482-4d87-8d74-6de00dde3fd4/volumes/kubernetes.io~projected/kube-api-access-kkxsf:{mountpoint:/var/lib/kubelet/pods/2af01093-4482-4d87-8d74-6de00dde3fd4/volumes/kubernetes.io~projected/kube-api-access-kkxsf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3725a48c-aeac-4da8-b9b8-8f8eb5790c00/volumes/kubernetes.io~projected/kube-api-access-85bss:{mountpoint:/var/lib/kubelet/pods/3725a48c-aeac-4da8-b9b8-8f8eb5790c00/volumes/kubernetes.io~projected/kube-api-access-85bss major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59555cdb-7ee8-4bd1-b100-a881c36ec6e9/volumes/kubernetes.io~projected/kube-api-access-p2svs:{mountpoint:/var/lib/kubelet/pods/59555cdb-7ee8-4bd1-b100-a881c36ec6e9/volumes/kubernetes.io~projected/kube-api-access-p2svs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adfb865-b3de-4b4e-b9c7-ae82f11456d6/volumes/kubernetes.io~projected/kube-api-access-4vsvh:{mountpoint:/var/lib/kubelet/pods/7adfb865-b3de-4b4e-b9c7-ae82f11456d6/volumes/kubernetes.io~projected/kube-api-access-4vsvh major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adfb865-b3de-4b4e-b9c7-ae82f11456d6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7adfb865-b3de-4b4e-b9c7-ae82f11456d6/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82c5066e-78ec-49ff-ba9f-bec5967833b9/volumes/kubernetes.io~projected/kube-api-access-j6hcn:{mountpoint:/var/lib/kubelet/pods/82c5066e-78ec-49ff-ba9f-bec5967833b9/volumes/kubernetes.io~projected/kube-api-access-j6hcn major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b62b472-60a7-442e-a38f-e7ca7b5172ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8b62b472-60a7-442e-a38f-e7ca7b5172ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volumes/kubernetes.io~projected/kube-api-access-jp5ts:{mountpoint:/var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volumes/kubernetes.io~projected/kube-api-access-jp5ts major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b73c5c00-4399-4546-a2fe-7f58e086e5a8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b73c5c00-4399-4546-a2fe-7f58e086e5a8/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5f49952-bd5a-49f7-9b11-79d3cfbceeff/volumes/kubernetes.io~projected/kube-api-access-tczgk:{mountpoint:/var/lib/kubelet/pods/c5f49952-bd5a-49f7-9b11-79d3cfbceeff/volumes/kubernetes.io~projected/kube-api-access-tczgk major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae/volumes/kubernetes.io~projected/kube-api-access-zw2cc:{mountpoint:/var/lib/kubelet/pods/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae/volumes/kubernetes.io~projected/kube-api-access-zw2cc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8dc270b-a9e6-46cc-b99c-9452eec6e013/volumes/kubernetes.io~projected/kube-api-access-g6wpf:{mountpoint:/var/lib/kubelet/pods/c8dc270b-a9e6-46cc-b99c-9452eec6e013/volumes/kubernetes.io~projected/kube-api-access-g6wpf major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf40282-bfe4-4279-8304-3173ec43dcf1/volumes/kubernetes.io~projected/kube-api-access-xbqmh:{mountpoint:/var/lib/kubelet/pods/cbf40282-bfe4-4279-8304-3173ec43dcf1/volumes/kubernetes.io~projected/kube-api-access-xbqmh major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~projected/kube-api-access-9fkv5:{mountpoint:/var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~projected/kube-api-access-9fkv5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b/volumes/kubernetes.io~projected/kube-api-access-5j9b8:{mountpoint:/var/lib/kubelet/pods/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b/volumes/kubernetes.io~projected/kube-api-access-5j9b8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6/volumes/kubernetes.io~projected/kube-api-access-cmrk7:{mountpoint:/var/lib/kubelet/pods/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6/volumes/kubernetes.io~projected/kube-api-access-cmrk7 major:0 minor:218 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f73e90bdd8a05fd506abf13413fed0ab82561b29ad8cdc1fdf62f9882d5181ce/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1db68ffbd4969ad4e6bd18441165d3c621435eca7848c21d8bc99bea92aa827b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f100f75f04cd1f6dd2352c629258aaf746d4dfd1e276bcc070c531fbda99b418/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b5c35cd36f2ad6865fc7b290e10dd2cc3022c0da18e6004207f3800e633c86e0/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/b183ce9f94dd518759f7f85dc5fe93c98c63857eeba9060cf88fc9907b0f05e7/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0fb512a2f4ff961eefc547940af0956af3f7eba6d3f1042567dba62b3587216c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fb01dc40c38b27fe6fd18a0209f45a94ca52bfbf870399cfd48a3f90e246a0b3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d07e628a55ea548fbdb87bf49cad7d1ba07c7d10b562d4395992a8f3433ad45e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/e0345a4a033c762f837e504a418c3759a2e98521daf77baf7b17d7b1b952999a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d28daf571cc76b3de3713ff552ff25543d62387cb007fee30a40e0c47339e280/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c07b2e39559ba6b7dc20b36d37f28fbcafbb66b85d4f813ce8750535969e518c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/204f283f56eec3983a62b5e8b55f8cef2a8e25f0ca52e61766be8f744b8dcc48/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/724bd725f6acb916bd16c743f1ed958b265829a94f7639d0ee02961988d740ee/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8f7385e7ec074b4ffc03e26d76e086908bb01b14e2a0e1e417c8e50f3d7b1d6d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a1af0d637ed2ed616ab297fcf3aafb3b9f69a0567730c54d369bbaf0495456f2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/42c4ab72915f7afda857e8df74d24c7f8b1f6df4b7aab72f96ddf2b8780f2181/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fff5e9c78c60d4f76e3f212fa6f9fef0e854434fb8e5e1334d55e1a44b4bb902/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/aa0fbdb79a330a94752df4aded7cf2e77133af0b3648ddb9d4431e3324cc3f06/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7956d39f24f4ce55c90474604d55b46a0e788385be6c36618e077a681a82a573/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fdcfabb11fcd02a9cf323a1cc8dc72449cd4c4b38fe290c4cefd18d234db2f9e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b90eae62dccec0e9ee333aad64b6017c3a0f0e5cdedfc514e11eb0cd11b54283/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/fec04fa0c7a17bda15572ce982cf03ee5b5139d4b292e2afc3dd2c491da4b80e/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/82c21ba33bae5ecc9af207debd5a6450662c289ed6f3ff1e98fcb837095ec24d/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/437f8c3accc36e8a1ba57107c525095b768416a5fbe552cad82a98e77fcfa049/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/fc9deb6bc21155a461e52ce88919b826fd3dcb2cd4d70b55bfb43a5602b52146/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/aaf681aab1c8af879529cfdf6293c684ba6e9d2886030a9efd0035fc233531fb/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/60d474abb325a0da1be0c31317f1bfa2ecb9e31c6c0111de6b77834faeedb469/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/cc659df8ffbf7d7e5e8bee39edd17e281b6fb08b1f6c2445162c4c13073849ba/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/e5c0e8e2161d3fc3f18e98e83abdded111f19352d67aa40a9086ba2aac07a565/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/451a7aed74c4c1cbc024c1fd275f4673a12e22543322cf99db9f313ba7b9b809/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/ab30367b078848a847c1a5de3291135778a0a1ca7358a1966ffe0ee442cc3423/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7048f3ca68352b8592ce36520ce54075e6a0e5a29ba98cc2d81eb0eaad34581b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/605092ac9c1f224ad5da14b54ec4a6f83a5f21c2be34aa02fb24fbde91624d75/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a22f73c4caa6aeb38bc8aec0522c1c43ffb554d555cfc6121bcbcd2097c8b808/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/4d2011a9d4c93e2b6f5f61b86df5bfe5fb6ed749612d9bfa96c72356ecdaa48a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a93737b709aa11045cae2e836b82378cf10a69391eaceec8e134d5a45afa106c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/d13dbc801a6755315f10f9851839b7240f8eab87823a5a807de5a20f30f19e84/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/03618a3e782b2d23a35bc21871674a938f99cd47850fa96c1299d8e3e38ab30a/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1f4be1b988af31d5a73897e74625e80ce21890013d34e799fcf6f2316a53cc80/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8dead26050096b653a6952284c932c024db9bdf432d6e4744020a640c601284d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f3cc4a06b3cf0e5c8c7b37fad38a5432d5bf7a798ff54d5e39279692d1b47323/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.526408    4468 manager.go:217] Machine: {Timestamp:2026-11-01 06:09:48.525951359 +0000 UTC m=+0.093758180 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099310 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27820165c162773ee030d57fda1962 SystemUUID:ec278201-65c1-6277-3ee0-30d57fda1962 BootID:916f0823-affe-4c60-8a5c-f801ce61d38a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbc09dc20bed3cc59b4fbe2ec36c1afd6f50fffe7118153a6e19c43174f5695c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbe25766fec9c99828763dbfbcc3258bf3c5f550d8b3de28318bc3c6a577796d/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59555cdb-7ee8-4bd1-b100-a881c36ec6e9/volumes/kubernetes.io~projected/kube-api-access-p2svs DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e9a30fd-cbc2-429c-bfab-788eef7ea27a/volumes/kubernetes.io~projected/kube-api-access-6qt59 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7adfb865-b3de-4b4e-b9c7-ae82f11456d6/volumes/kubernetes.io~projected/kube-api-access-4vsvh DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b/volumes/kubernetes.io~projected/kube-api-access-5j9b8 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fcf1a29e8c552c7b9d7fd40078d9e52cf0010972152dfe2ae519d056399a497/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8014a0eff340bf9018966cc50874ff1d6ca3f62a424f4627dea30ebd687e994/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b62b472-60a7-442e-a38f-e7ca7b5172ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0b051ca1a07691fe44806142980d32546d4a7e8098640289ece763b24b877eb/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b73c5c00-4399-4546-a2fe-7f58e086e5a8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b3a8bc2e0184c576609c024b5f793d51619a3bff6c27bd4981f7bf1abe06c6c/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82c5066e-78ec-49ff-ba9f-bec5967833b9/volumes/kubernetes.io~projected/kube-api-access-j6hcn DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6/volumes/kubernetes.io~projected/kube-api-access-cmrk7 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e165aa3a7b0383e8a1a7f190eb9096410e9c6090f2eaf34e72ddb76c320190d3/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5f49952-bd5a-49f7-9b11-79d3cfbceeff/volumes/kubernetes.io~projected/kube-api-access-tczgk DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d87afd64cc5d8c7c725241ae8f3ae62b2983e1782e48de9dd451efe1e07f104d/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3725a48c-aeac-4da8-b9b8-8f8eb5790c00/volumes/kubernetes.io~projected/kube-api-access-85bss DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~projected/kube-api-access-9fkv5 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1dd22b57-4397-461a-ba98-57179378a67d/volumes/kubernetes.io~projected/kube-api-access-qcf54 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c2bdf5ef05d7a551872f8ed3bbdad91763dd77d3b81630e3ca20fb45aeb3176/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c05d29fd39519e18cca5b14996382c489ac5d70e9225954e44bb0ada9934706d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae/volumes/kubernetes.io~projected/kube-api-access-zw2cc DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6341d0915f68f56f72dca8c8dd8a7c43961340450d238d1255492d79952627b6/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e041a721-484d-4b8b-8611-fa6d4d8c4602/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a529fecd-b712-470e-87e4-56fa923ad3b9/volumes/kubernetes.io~projected/kube-api-access-jp5ts DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34cc0f4a60c3eee25e027eb3acbdee7191c28414073a1565db84aa82c2b647ee/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8dc270b-a9e6-46cc-b99c-9452eec6e013/volumes/kubernetes.io~projected/kube-api-access-g6wpf DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2af01093-4482-4d87-8d74-6de00dde3fd4/volumes/kubernetes.io~projected/kube-api-access-kkxsf DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7adfb865-b3de-4b4e-b9c7-ae82f11456d6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf40282-bfe4-4279-8304-3173ec43dcf1/volumes/kubernetes.io~projected/kube-api-access-xbqmh DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:6341d0915f68f56 MacAddress:7e:a8:e6:6a:81:50 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:52:07:da:3f:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:b4:4c:6d:2f:90 Speed:0 Mtu:8901} {Name:c0b051ca1a07691 MacAddress:da:ee:39:2d:6b:c8 Speed:10000 Mtu:8901} {Name:d3736e1e6359b6c MacAddress:8a:7d:07:a6:62:cc Speed:10000 Mtu:8901} {Name:d87afd64cc5d8c7 MacAddress:8e:45:50:2f:3d:96 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:52:07:da:3f:4d Speed:0 Mtu:9001} {Name:fbe25766fec9c99 MacAddress:62:45:44:48:04:de Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:82:19:32:56:86:4e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:e9:d4:e6:ec:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.526717    4468 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.526762    4468 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527032    4468 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527057    4468 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-21.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527538    4468 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527557    4468 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527573    4468 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527592    4468 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527705    4468 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.527956    4468 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.528102    4468 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.528146    4468 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.528163    4468 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.528180    4468 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.528194    4468 kubelet.go:405] "Adding apiserver pod source"
Nov 01 06:09:48.530927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.528210    4468 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 06:09:48.539839 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.539815    4468 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 06:09:48.539839 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.539837    4468 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 06:09:48.540504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.540461    4468 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 06:09:48.542357 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542343    4468 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 06:09:48.542420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542358    4468 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 06:09:48.542548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542455    4468 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 06:09:48.542690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542679    4468 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 06:09:48.542726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542709    4468 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 06:09:48.542807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542799    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 06:09:48.542845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542817    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 06:09:48.542845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542827    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 06:09:48.542845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542833    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 06:09:48.542845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542839    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 06:09:48.542845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542845    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542851    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542857    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542864    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542870    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542890    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542900    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542928    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 06:09:48.542978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.542937    4468 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 06:09:48.546332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.546317    4468 server.go:1263] "Started kubelet"
Nov 01 06:09:48.546413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.546383    4468 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 06:09:48.546512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.546423    4468 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 06:09:48.546562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.546542    4468 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.546921    4468 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.547640    4468 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549028    4468 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549069    4468 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549103    4468 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549117    4468 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.549166    4468 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549212    4468 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549128    4468 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:10 +0000 UTC" deadline="2028-04-11 22:27:59.566456402 +0000 UTC"
Nov 01 06:09:48.551047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.549331    4468 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12664h18m11.017130102s"
Nov 01 06:09:48.546942 ip-10-0-138-21 systemd[1]: Started Kubernetes Kubelet.
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552549    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3725a48c-aeac-4da8-b9b8-8f8eb5790c00" volumeName="kubernetes.io/configmap/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-iptables-alerter-script" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552621    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a529fecd-b712-470e-87e4-56fa923ad3b9" volumeName="kubernetes.io/projected/a529fecd-b712-470e-87e4-56fa923ad3b9-kube-api-access-jp5ts" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552639    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e041a721-484d-4b8b-8611-fa6d4d8c4602" volumeName="kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-tmp" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552654    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dd22b57-4397-461a-ba98-57179378a67d" volumeName="kubernetes.io/configmap/1dd22b57-4397-461a-ba98-57179378a67d-serviceca" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552701    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82c5066e-78ec-49ff-ba9f-bec5967833b9" volumeName="kubernetes.io/configmap/82c5066e-78ec-49ff-ba9f-bec5967833b9-config-volume" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552717    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82c5066e-78ec-49ff-ba9f-bec5967833b9" volumeName="kubernetes.io/empty-dir/82c5066e-78ec-49ff-ba9f-bec5967833b9-tmp-dir" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552746    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a529fecd-b712-470e-87e4-56fa923ad3b9" volumeName="kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-config" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552760    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73c5c00-4399-4546-a2fe-7f58e086e5a8" volumeName="kubernetes.io/secret/b73c5c00-4399-4546-a2fe-7f58e086e5a8-agent-certs" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552865    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf40282-bfe4-4279-8304-3173ec43dcf1" volumeName="kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552883    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf40282-bfe4-4279-8304-3173ec43dcf1" volumeName="kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552898    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dd22b57-4397-461a-ba98-57179378a67d" volumeName="kubernetes.io/projected/1dd22b57-4397-461a-ba98-57179378a67d-kube-api-access-qcf54" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552918    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2af01093-4482-4d87-8d74-6de00dde3fd4" volumeName="kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552933    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b" volumeName="kubernetes.io/projected/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kube-api-access-5j9b8" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552953    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2af01093-4482-4d87-8d74-6de00dde3fd4" volumeName="kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552969    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b62b472-60a7-442e-a38f-e7ca7b5172ec" volumeName="kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.552988    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2af01093-4482-4d87-8d74-6de00dde3fd4" volumeName="kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-binary-copy" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553002    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3725a48c-aeac-4da8-b9b8-8f8eb5790c00" volumeName="kubernetes.io/projected/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-kube-api-access-85bss" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553016    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82c5066e-78ec-49ff-ba9f-bec5967833b9" volumeName="kubernetes.io/projected/82c5066e-78ec-49ff-ba9f-bec5967833b9-kube-api-access-j6hcn" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553036    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a529fecd-b712-470e-87e4-56fa923ad3b9" volumeName="kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-env-overrides" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553049    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73c5c00-4399-4546-a2fe-7f58e086e5a8" volumeName="kubernetes.io/configmap/b73c5c00-4399-4546-a2fe-7f58e086e5a8-konnectivity-ca" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553067    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c618c3d8-5fa0-4e2e-a222-81b672c9b0ae" volumeName="kubernetes.io/projected/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-kube-api-access-zw2cc" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553080    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf40282-bfe4-4279-8304-3173ec43dcf1" volumeName="kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553094    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5f49952-bd5a-49f7-9b11-79d3cfbceeff" volumeName="kubernetes.io/empty-dir/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-data-volume" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553111    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9a30fd-cbc2-429c-bfab-788eef7ea27a" volumeName="kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-daemon-config" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553125    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9a30fd-cbc2-429c-bfab-788eef7ea27a" volumeName="kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cni-binary-copy" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553144    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adfb865-b3de-4b4e-b9c7-ae82f11456d6" volumeName="kubernetes.io/secret/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-cert" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553158    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a529fecd-b712-470e-87e4-56fa923ad3b9" volumeName="kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553172    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5f49952-bd5a-49f7-9b11-79d3cfbceeff" volumeName="kubernetes.io/projected/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-api-access-tczgk" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553197    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8dc270b-a9e6-46cc-b99c-9452eec6e013" volumeName="kubernetes.io/projected/c8dc270b-a9e6-46cc-b99c-9452eec6e013-kube-api-access-g6wpf" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553217    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf40282-bfe4-4279-8304-3173ec43dcf1" volumeName="kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553231    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e041a721-484d-4b8b-8611-fa6d4d8c4602" volumeName="kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-tuned" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553248    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2af01093-4482-4d87-8d74-6de00dde3fd4" volumeName="kubernetes.io/projected/2af01093-4482-4d87-8d74-6de00dde3fd4-kube-api-access-kkxsf" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553261    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59555cdb-7ee8-4bd1-b100-a881c36ec6e9" volumeName="kubernetes.io/empty-dir/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-tmp-dir" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553284    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59555cdb-7ee8-4bd1-b100-a881c36ec6e9" volumeName="kubernetes.io/projected/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-kube-api-access-p2svs" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553301    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adfb865-b3de-4b4e-b9c7-ae82f11456d6" volumeName="kubernetes.io/projected/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-kube-api-access-4vsvh" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553313    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a529fecd-b712-470e-87e4-56fa923ad3b9" volumeName="kubernetes.io/secret/a529fecd-b712-470e-87e4-56fa923ad3b9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553332    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e041a721-484d-4b8b-8611-fa6d4d8c4602" volumeName="kubernetes.io/projected/e041a721-484d-4b8b-8611-fa6d4d8c4602-kube-api-access-9fkv5" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553346    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f320aeca-d424-42de-bfbc-c0ef0eaa2bc6" volumeName="kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553359    4468 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9a30fd-cbc2-429c-bfab-788eef7ea27a" volumeName="kubernetes.io/projected/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-kube-api-access-6qt59" seLinuxMountContext=""
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553371    4468 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553378    4468 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553885    4468 factory.go:55] Registering systemd factory
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.553907    4468 factory.go:223] Registration of the systemd container factory successfully
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.555216    4468 factory.go:153] Registering CRI-O factory
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.555232    4468 factory.go:223] Registration of the crio container factory successfully
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.555314    4468 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.555335    4468 factory.go:103] Registering Raw factory
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.555350    4468 manager.go:1199] Started watching for new ooms in manager
Nov 01 06:09:48.556504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.555784    4468 manager.go:319] Starting recovery of all containers
Nov 01 06:09:48.580625 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.580576    4468 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 06:09:48.582220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.582194    4468 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 06:09:48.582316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.582224    4468 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 06:09:48.582316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.582248    4468 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 06:09:48.582316 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.582291    4468 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 06:09:48.603904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.603869    4468 generic.go:356] "Generic (PLEG): container finished" podID="23d718e8dfa8061bd8b70748a702b624" containerID="2cc3ac5b933865f4a479b98eb090a7eab1d5b96c556cc45a7811a5a342e8fe9c" exitCode=0
Nov 01 06:09:48.607337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.607308    4468 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="3be5fcc3b7eb6b6d3e2562fd8f93ee619aa615fbf99b9806f4aae430ab2063af" exitCode=0
Nov 01 06:09:48.607337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.607338    4468 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="7f1e2d682260caffce568fd9be95e764dd688a419dc30611b52c5a0bf9a97910" exitCode=0
Nov 01 06:09:48.607337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.607346    4468 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="b4a1e8cc05d4651f0a22e379cff971febb5bf1fb275c814819367fa6f4a9a967" exitCode=0
Nov 01 06:09:48.607569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.607357    4468 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="b68e016ce9786ebc5e0e757cc6fdcca60377110b617357a745216a224d30b24a" exitCode=0
Nov 01 06:09:48.607569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.607366    4468 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="4f8b8668a942c2509efb69e70e74114a54bd38bfa1998830e80f541797076bfe" exitCode=0
Nov 01 06:09:48.607569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.607373    4468 generic.go:356] "Generic (PLEG): container finished" podID="2af01093-4482-4d87-8d74-6de00dde3fd4" containerID="6f9d2ca375d0c6da3a7cb38fa63ed90fcd97f88eadb9b91df122692f81fa6fed" exitCode=0
Nov 01 06:09:48.624828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.624773    4468 manager.go:324] Recovery completed
Nov 01 06:09:48.637849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.637827    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.639151 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.639135    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.639216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.639164    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.639216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.639175    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.640578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640560    4468 cpu_manager.go:226] "Starting" policy="none"
Nov 01 06:09:48.640578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640576    4468 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 06:09:48.640703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640579    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.640703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640594    4468 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 06:09:48.640792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640743    4468 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 06:09:48.640792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640759    4468 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 06:09:48.640792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640782    4468 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 06:09:48.640792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640790    4468 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 06:09:48.640974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640798    4468 policy_none.go:50] "Start"
Nov 01 06:09:48.640974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640810    4468 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 06:09:48.640974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.640823    4468 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 06:09:48.641103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.641003    4468 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 06:09:48.641103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.641014    4468 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 06:09:48.641103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.641022    4468 policy_none.go:44] "Start"
Nov 01 06:09:48.641497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.641483    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.641562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.641512    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.641562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.641526    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.645377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.645340    4468 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 06:09:48.647908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.647891    4468 manager.go:341] "Starting Device Plugin manager"
Nov 01 06:09:48.647994 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.647917    4468 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 06:09:48.647994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.647929    4468 server.go:87] "Starting device plugin registration server"
Nov 01 06:09:48.648200 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648187    4468 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 06:09:48.648259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648199    4468 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 06:09:48.648312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648291    4468 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 06:09:48.648409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648378    4468 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 06:09:48.648586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648418    4468 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 06:09:48.648586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648425    4468 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 06:09:48.648586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.648494    4468 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T06:09:48.648407199Z","UUID":"9a372cb8-30d9-4dc2-ab47-cf274f08ef75","Handler":null,"Name":"","Endpoint":""}
Nov 01 06:09:48.649819 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.649799    4468 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 06:09:48.649918 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.649836    4468 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-21.us-west-1.compute.internal\" not found"
Nov 01 06:09:48.650862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.650844    4468 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 06:09:48.650948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.650869    4468 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 06:09:48.683533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.683026    4468 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"]
Nov 01 06:09:48.683665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.683627    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.684712 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.684696    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.684767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.684727    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.684767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.684739    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.685001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.684991    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.685140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.685127    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.685978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.685958    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.686081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.685989    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.686081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686004    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.686081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686020    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.686081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686042    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.686081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686052    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.686231 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.686192    4468 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.686293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686265    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"5a65f0fc8a51566f4a4d62ed9c49d84d","Type":"ContainerStarted","Data":"f91ae8c39704e4c64c74ce07d693c899305b7766a361617b5fa753bfd11af070"}
Nov 01 06:09:48.686323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686299    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"5a65f0fc8a51566f4a4d62ed9c49d84d","Type":"ContainerStarted","Data":"dc8e887ab87c4ba6e449b28ea4872510376309f3d883ad214092253580e67d42"}
Nov 01 06:09:48.686323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686312    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"23d718e8dfa8061bd8b70748a702b624","Type":"ContainerStarted","Data":"566fc3810bf3f902d06e9d31e17600651af18ba71aa59a6475c11adca3dd5e1c"}
Nov 01 06:09:48.686380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686323    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"23d718e8dfa8061bd8b70748a702b624","Type":"ContainerDied","Data":"2cc3ac5b933865f4a479b98eb090a7eab1d5b96c556cc45a7811a5a342e8fe9c"}
Nov 01 06:09:48.686380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686341    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.686380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.686337    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal" event={"ID":"23d718e8dfa8061bd8b70748a702b624","Type":"ContainerStarted","Data":"a411dbd311cb54b939e0b4232759d7666a534ad41c55d6fbd9b57489e6a3f4c3"}
Nov 01 06:09:48.687210 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.687194    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.687251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.687219    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.687251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.687229    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.687357 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.687346    4468 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-21.us-west-1.compute.internal\" not found" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.744959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.744925    4468 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:48.746007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.745989    4468 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:48.746089 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.746076    4468 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 06:09:48.746140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.746093    4468 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:48.749095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.749080    4468 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 06:09:48.749358 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.749339    4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.752720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.752702    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 06:09:48.752811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.752730    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 06:09:48.752811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.752740    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 06:09:48.752811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.752773    4468 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.795887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.795847    4468 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 06:09:48.836340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.836310    4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 06:09:48.836512 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.836371    4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.836512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.836388    4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.846557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.846530    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.846610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.846565    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.846610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.846592    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a65f0fc8a51566f4a4d62ed9c49d84d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"5a65f0fc8a51566f4a4d62ed9c49d84d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.887845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.887775    4468 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.887969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.887851    4468 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.888435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.888417    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeNotReady"
Nov 01 06:09:48.888537 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.888445    4468 setters.go:546] "Node became not ready" node="ip-10-0-138-21.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T06:09:48Z","lastTransitionTime":"2026-11-01T06:09:48Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 06:09:48.893555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.893540    4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 06:09:48.893600 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:48.893584    4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.904788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.904769    4468 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-21.us-west-1.compute.internal" is owned by different node (oldNodeID="44ffcc32-035a-4f94-8a33-8d04b7d22c34", newNodeID=""), cleaning up...
Nov 01 06:09:48.947326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.947293    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.947437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.947330    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.947437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.947350    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a65f0fc8a51566f4a4d62ed9c49d84d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"5a65f0fc8a51566f4a4d62ed9c49d84d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.947437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.947398    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a65f0fc8a51566f4a4d62ed9c49d84d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"5a65f0fc8a51566f4a4d62ed9c49d84d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.947437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.947412    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:48.947601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:48.947416    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23d718e8dfa8061bd8b70748a702b624-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" (UID: \"23d718e8dfa8061bd8b70748a702b624\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:49.153882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.153813    4468 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-21.us-west-1.compute.internal" event="NodeReady"
Nov 01 06:09:49.154014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.153890    4468 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 06:09:49.529449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.529371    4468 apiserver.go:52] "Watching apiserver"
Nov 01 06:09:49.618786 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.618757    4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:49.618969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.618900    4468 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:49.634219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.634081    4468 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 06:09:49.635043 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.635016    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-bq58b","openshift-dns/node-resolver-sdt2h","openshift-insights/periodic-gathering-dn84m-krqpd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q","openshift-ingress-canary/ingress-canary-whgfv","openshift-multus/network-metrics-daemon-l6492","openshift-multus/multus-pvbrt","openshift-ovn-kubernetes/ovnkube-node-2rnnc","openshift-dns/dns-default-99jhh","openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp","openshift-network-diagnostics/network-check-target-7llbh","openshift-cluster-node-tuning-operator/tuned-8rjvr","kube-system/global-pull-secret-syncer-v48xf","openshift-insights/insights-runtime-extractor-q8k87","kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal","openshift-image-registry/node-ca-r5fnn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal","kube-system/konnectivity-agent-w6l2n","openshift-network-operator/iptables-alerter-pr9kp"]
Nov 01 06:09:49.636771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.636750    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:49.639547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.639528    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.639973 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.639952    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.651023 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.650999    4468 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 06:09:49.651147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651133    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cni-binary-copy\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651157    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-conf-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651173    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-slash\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651187    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-systemd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651203    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-socket-dir-parent\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651270    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-netns\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651302    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-kubelet\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651317    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-hostroot\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651334    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-ovn\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651354    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-node-log\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651395    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-systemd\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.651438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651420    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.651507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651498    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651520    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cnibin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651570 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651536    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-etc-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651573    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-config\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651599    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-script-lib\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.651672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651647    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.651782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651677    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.651782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651710    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-multus-certs\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.651782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651744    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-dbus\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.651782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651766    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wpf\" (UniqueName: \"kubernetes.io/projected/c8dc270b-a9e6-46cc-b99c-9452eec6e013-kube-api-access-g6wpf\") pod \"migrator-66df5d496b-5n8jp\" (UID: \"c8dc270b-a9e6-46cc-b99c-9452eec6e013\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:49.651782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651781    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-cnibin\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.651917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651820    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:49.651917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651909    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-sys\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.651970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651925    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-host\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.651970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.651960    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-tuned\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652023 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652003    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.652065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652050    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-tuned\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652062    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c5066e-78ec-49ff-ba9f-bec5967833b9-tmp-dir\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.652123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652093    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-crio-socket\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.652151 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652132    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-data-volume\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.652182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652150    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c5066e-78ec-49ff-ba9f-bec5967833b9-tmp-dir\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.652182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652151    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-bin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.652237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652182    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-systemd-units\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.652265 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652240    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-tmp\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652265 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652241    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-data-volume\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.652322 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652282    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.652322 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652306    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e041a721-484d-4b8b-8611-fa6d4d8c4602-tmp\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652330    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.652374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652346    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-run\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652386    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-var-lib-kubelet\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652405    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-cert\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:49.652506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652454    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.652542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652507    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.652571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652539    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-device-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.652571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652563    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf54\" (UniqueName: \"kubernetes.io/projected/1dd22b57-4397-461a-ba98-57179378a67d-kube-api-access-qcf54\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.652626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652608    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-kubelet\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.652653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652631    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-kubernetes\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652650    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.652705 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652677    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsvh\" (UniqueName: \"kubernetes.io/projected/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-kube-api-access-4vsvh\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:49.652768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652703    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.652768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652720    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-k8s-cni-cncf-io\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.652768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652734    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-bin\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.652768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652755    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5ts\" (UniqueName: \"kubernetes.io/projected/a529fecd-b712-470e-87e4-56fa923ad3b9-kube-api-access-jp5ts\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.652873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652790    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.652873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652804    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.652873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652829    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-sys-fs\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.652873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652853    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9b8\" (UniqueName: \"kubernetes.io/projected/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kube-api-access-5j9b8\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.652981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652888    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2cc\" (UniqueName: \"kubernetes.io/projected/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-kube-api-access-zw2cc\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:49.652981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652905    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dd22b57-4397-461a-ba98-57179378a67d-serviceca\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.652981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652919    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt59\" (UniqueName: \"kubernetes.io/projected/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-kube-api-access-6qt59\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.652981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652936    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysconfig\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.653085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.652982    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-etc-kubernetes\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.653085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653018    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-kubelet-config\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.653085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653065    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-netd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.653163 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653093    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-iptables-alerter-script\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.653163 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653143    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b73c5c00-4399-4546-a2fe-7f58e086e5a8-konnectivity-ca\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:49.653163 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653159    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.653238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653174    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-tmp-dir\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.653238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653191    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-multus\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.653238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653215    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.653319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653264    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-tmp-dir\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.653319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653297    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.653319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653316    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.653401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653334    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bss\" (UniqueName: \"kubernetes.io/projected/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-kube-api-access-85bss\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.653401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653350    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.653401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653378    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b73c5c00-4399-4546-a2fe-7f58e086e5a8-agent-certs\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:49.653503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653409    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-hosts-file\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.653503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653459    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-system-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.653503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653499    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-daemon-config\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.653586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653551    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-var-lib-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.653586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653573    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-socket-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.653640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653588    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-registration-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.653640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653604    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a529fecd-b712-470e-87e4-56fa923ad3b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.653697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653635    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxsf\" (UniqueName: \"kubernetes.io/projected/2af01093-4482-4d87-8d74-6de00dde3fd4-kube-api-access-kkxsf\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.653697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653674    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-modprobe-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.653748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653694    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkv5\" (UniqueName: \"kubernetes.io/projected/e041a721-484d-4b8b-8611-fa6d4d8c4602-kube-api-access-9fkv5\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.653748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653731    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82c5066e-78ec-49ff-ba9f-bec5967833b9-config-volume\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.653802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653761    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.653802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653788    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:49.653888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653815    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hcn\" (UniqueName: \"kubernetes.io/projected/82c5066e-78ec-49ff-ba9f-bec5967833b9-kube-api-access-j6hcn\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.653888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653835    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.653888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653865    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-system-cni-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.653966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653947    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-conf\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.653994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653971    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-lib-modules\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.653994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.653988    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2svs\" (UniqueName: \"kubernetes.io/projected/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-kube-api-access-p2svs\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.654100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.654089    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-netns\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.654131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.654114    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-log-socket\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.654162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.654129    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-env-overrides\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.654195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.654163    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-host-slash\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.654195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.654178    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dd22b57-4397-461a-ba98-57179378a67d-host\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.654250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.654194    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczgk\" (UniqueName: \"kubernetes.io/projected/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-api-access-tczgk\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.678160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.678138    4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 06:09:49.678254 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:49.678186    4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:49.681231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.681217    4468 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 06:09:49.681267 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:49.681252    4468 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-21.us-west-1.compute.internal"
Nov 01 06:09:49.688417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.688397    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 06:09:49.688493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.688421    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.688548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.688535    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.689941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689854    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.689941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689868    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.689941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689881    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 06:09:49.689941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689900    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 06:09:49.689941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689929    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 06:09:49.689941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689939    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rlspf\""
Nov 01 06:09:49.690190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689865    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 06:09:49.690190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689945    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnlbn\""
Nov 01 06:09:49.690190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689974    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.690190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.689989    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.690190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690082    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 06:09:49.690377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690211    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.690377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690282    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.690377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690302    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.690377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690308    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.690377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690338    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 06:09:49.690377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690346    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 06:09:49.690636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690622    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 06:09:49.690828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690812    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 06:09:49.690871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690816    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.690976 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.690964    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78q7p\""
Nov 01 06:09:49.691049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691040    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 06:09:49.691089 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691068    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 06:09:49.691136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691108    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vsdrw\""
Nov 01 06:09:49.691178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691139    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 06:09:49.691178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691151    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8r6tp\""
Nov 01 06:09:49.691178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691140    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wss87\""
Nov 01 06:09:49.691683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691658    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 06:09:49.691786 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691772    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 06:09:49.691884 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691865    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnjvc\""
Nov 01 06:09:49.691937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691888    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-config\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.691987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.691870    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cni-binary-copy\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.692145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.692127    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.694797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.694599    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-env-overrides\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.694797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.694628    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.694797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.694646    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1dd22b57-4397-461a-ba98-57179378a67d-serviceca\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.694797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.694748    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b62b472-60a7-442e-a38f-e7ca7b5172ec-original-pull-secret\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.694797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.694762    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-iptables-alerter-script\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.695041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.694774    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b73c5c00-4399-4546-a2fe-7f58e086e5a8-konnectivity-ca\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:49.695209 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.695187    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2af01093-4482-4d87-8d74-6de00dde3fd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.695300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.695192    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-daemon-config\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.695300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.695249    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b73c5c00-4399-4546-a2fe-7f58e086e5a8-agent-certs\") pod \"konnectivity-agent-w6l2n\" (UID: \"b73c5c00-4399-4546-a2fe-7f58e086e5a8\") " pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:49.695377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.695343    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a529fecd-b712-470e-87e4-56fa923ad3b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.702444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.702420    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a529fecd-b712-470e-87e4-56fa923ad3b9-ovnkube-script-lib\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.727878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.727847    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdp5j\""
Nov 01 06:09:49.727878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.727851    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnk2x\""
Nov 01 06:09:49.730851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.730826    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.730851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.730832    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.731005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.730914    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwtj8\""
Nov 01 06:09:49.735665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.735649    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.735733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.735709    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h2lht\""
Nov 01 06:09:49.736849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736836    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.736849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736846    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.736993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736854    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m98z\""
Nov 01 06:09:49.736993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736865    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.736993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736954    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 06:09:49.736993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736963    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.736993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.736975    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 06:09:49.737150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737033    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 06:09:49.737150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737073    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgqp9\""
Nov 01 06:09:49.737230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737182    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8pggw\""
Nov 01 06:09:49.737767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737748    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.737902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737805    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 06:09:49.737902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737822    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 06:09:49.737902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737866    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 06:09:49.737902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737877    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 06:09:49.738054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737908    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.738054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.737968    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 06:09:49.738134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.738128    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jp6zc\""
Nov 01 06:09:49.738134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.738131    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9gkn9\""
Nov 01 06:09:49.738659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.738644    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wr\""
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.740801    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741286    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bss\" (UniqueName: \"kubernetes.io/projected/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-kube-api-access-85bss\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741331    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt59\" (UniqueName: \"kubernetes.io/projected/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-kube-api-access-6qt59\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741339    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2cc\" (UniqueName: \"kubernetes.io/projected/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-kube-api-access-zw2cc\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741344    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrk7\" (UniqueName: \"kubernetes.io/projected/f320aeca-d424-42de-bfbc-c0ef0eaa2bc6-kube-api-access-cmrk7\") pod \"network-check-target-7llbh\" (UID: \"f320aeca-d424-42de-bfbc-c0ef0eaa2bc6\") " pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741410    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf54\" (UniqueName: \"kubernetes.io/projected/1dd22b57-4397-461a-ba98-57179378a67d-kube-api-access-qcf54\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741583    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.741643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741334    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5ts\" (UniqueName: \"kubernetes.io/projected/a529fecd-b712-470e-87e4-56fa923ad3b9-kube-api-access-jp5ts\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.742085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741815    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczgk\" (UniqueName: \"kubernetes.io/projected/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-api-access-tczgk\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.742085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741905    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wpf\" (UniqueName: \"kubernetes.io/projected/c8dc270b-a9e6-46cc-b99c-9452eec6e013-kube-api-access-g6wpf\") pod \"migrator-66df5d496b-5n8jp\" (UID: \"c8dc270b-a9e6-46cc-b99c-9452eec6e013\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5n8jp"
Nov 01 06:09:49.742085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.741925    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6hcn\" (UniqueName: \"kubernetes.io/projected/82c5066e-78ec-49ff-ba9f-bec5967833b9-kube-api-access-j6hcn\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.742240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742123    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsvh\" (UniqueName: \"kubernetes.io/projected/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-kube-api-access-4vsvh\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:49.742240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742173    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxsf\" (UniqueName: \"kubernetes.io/projected/2af01093-4482-4d87-8d74-6de00dde3fd4-kube-api-access-kkxsf\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.742240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742182    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkv5\" (UniqueName: \"kubernetes.io/projected/e041a721-484d-4b8b-8611-fa6d4d8c4602-kube-api-access-9fkv5\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.742385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742270    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9b8\" (UniqueName: \"kubernetes.io/projected/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kube-api-access-5j9b8\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.742455 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742434    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2svs\" (UniqueName: \"kubernetes.io/projected/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-kube-api-access-p2svs\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.742635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742619    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adfb865-b3de-4b4e-b9c7-ae82f11456d6-cert\") pod \"ingress-canary-whgfv\" (UID: \"7adfb865-b3de-4b4e-b9c7-ae82f11456d6\") " pod="openshift-ingress-canary/ingress-canary-whgfv"
Nov 01 06:09:49.742799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.742783    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.744120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.744103    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") pod \"periodic-gathering-dn84m-krqpd\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") " pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:09:49.744169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.744156    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82c5066e-78ec-49ff-ba9f-bec5967833b9-config-volume\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.754631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754604    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-socket-dir-parent\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754632    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-netns\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754648    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-kubelet\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754661    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-hostroot\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754676    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-ovn\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754689    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-node-log\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754704    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-systemd\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754717    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-kubelet\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754728    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754717    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-netns\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754753    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.754760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754756    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-ovn\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754756    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-socket-dir-parent\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754760    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-node-log\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754794    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754775    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-hostroot\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754775    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cnibin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754805    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-systemd\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754823    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-cnibin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754823    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754873    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-etc-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754911    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-etc-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754953    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754986    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-multus-certs\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.754999    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755014    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-dbus\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755040    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-cnibin\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755062    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-sys\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755098    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-multus-certs\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755096    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-cnibin\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755105    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-dbus\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755113    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-sys\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755115    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-host\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755141    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-host\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755143    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755168    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-crio-socket\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755195    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-bin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755208    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-crio-socket\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.755208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755212    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-systemd-units\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755236    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-bin\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755239    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-systemd-units\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:49.755247    4468 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755252    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755279    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-run\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755309    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-var-lib-kubelet\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:49.755331    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:50.255314902 +0000 UTC m=+1.823121741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755342    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-run\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755350    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755365    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-device-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:49.755370    4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755380    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-kubelet\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755363    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-var-lib-kubelet\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755405    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-kubelet\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755411    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-device-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:49.755420    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:50.255408698 +0000 UTC m=+1.823215506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755441    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-kubernetes\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755456    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-kubernetes\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755457    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755439    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755506    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-k8s-cni-cncf-io\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755527    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-bin\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755530    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-run-k8s-cni-cncf-io\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755542    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-sys-fs\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755557    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-bin\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755561    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755559    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysconfig\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755584    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysconfig\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755595    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-sys-fs\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755602    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-etc-kubernetes\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755585    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-etc-kubernetes\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755622    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-kubelet-config\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755636    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-netd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755651    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-multus\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755664    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755680    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755691    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-host-var-lib-cni-multus\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755700    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755699    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-cni-netd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755704    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755725    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755732    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-hosts-file\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755755    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-system-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755769    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-var-lib-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755733    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b62b472-60a7-442e-a38f-e7ca7b5172ec-kubelet-config\") pod \"global-pull-secret-syncer-v48xf\" (UID: \"8b62b472-60a7-442e-a38f-e7ca7b5172ec\") " pod="kube-system/global-pull-secret-syncer-v48xf"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755784    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-socket-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755803    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-system-cni-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755807    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59555cdb-7ee8-4bd1-b100-a881c36ec6e9-hosts-file\") pod \"node-resolver-sdt2h\" (UID: \"59555cdb-7ee8-4bd1-b100-a881c36ec6e9\") " pod="openshift-dns/node-resolver-sdt2h"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755817    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-var-lib-openvswitch\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755821    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755828    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-socket-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755845    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-registration-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755865    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-modprobe-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755886    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755895    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b-registration-dir\") pod \"aws-ebs-csi-driver-node-hlm2q\" (UID: \"e5d043f5-0ea8-4af9-8ec2-f7abdbc4f50b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlm2q"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755903    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755925    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-system-cni-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755941    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-conf\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755955    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-lib-modules\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755972    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-modprobe-d\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755974    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-netns\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756023    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-log-socket\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756060    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-host-slash\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.755981    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2af01093-4482-4d87-8d74-6de00dde3fd4-system-cni-dir\") pod \"multus-additional-cni-plugins-bq58b\" (UID: \"2af01093-4482-4d87-8d74-6de00dde3fd4\") " pod="openshift-multus/multus-additional-cni-plugins-bq58b"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756087    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dd22b57-4397-461a-ba98-57179378a67d-host\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756092    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-log-socket\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756097    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-etc-sysctl-conf\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756063    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e041a721-484d-4b8b-8611-fa6d4d8c4602-lib-modules\") pod \"tuned-8rjvr\" (UID: \"e041a721-484d-4b8b-8611-fa6d4d8c4602\") " pod="openshift-cluster-node-tuning-operator/tuned-8rjvr"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756113    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-conf-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756124    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1dd22b57-4397-461a-ba98-57179378a67d-host\") pod \"node-ca-r5fnn\" (UID: \"1dd22b57-4397-461a-ba98-57179378a67d\") " pod="openshift-image-registry/node-ca-r5fnn"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756137    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-slash\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756146    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9a30fd-cbc2-429c-bfab-788eef7ea27a-multus-conf-dir\") pod \"multus-pvbrt\" (UID: \"1e9a30fd-cbc2-429c-bfab-788eef7ea27a\") " pod="openshift-multus/multus-pvbrt"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756163    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-systemd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756166    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-run-netns\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756113    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3725a48c-aeac-4da8-b9b8-8f8eb5790c00-host-slash\") pod \"iptables-alerter-pr9kp\" (UID: \"3725a48c-aeac-4da8-b9b8-8f8eb5790c00\") " pod="openshift-network-operator/iptables-alerter-pr9kp"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756190    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-host-slash\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756195    4468 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 06:09:49.756260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756220    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a529fecd-b712-470e-87e4-56fa923ad3b9-run-systemd\") pod \"ovnkube-node-2rnnc\" (UID: \"a529fecd-b712-470e-87e4-56fa923ad3b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:49.758524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.756394    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:49.758524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.758372    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c618c3d8-5fa0-4e2e-a222-81b672c9b0ae-metrics-certs\") pod \"network-metrics-daemon-l6492\" (UID: \"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae\") " pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:49.937015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:49.936974    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6492"
Nov 01 06:09:50.259939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:50.259857    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:50.259939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:50.259901    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:50.260161 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:50.260011    4468 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:50.260161 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:50.260047    4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:50.260161 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:50.260083    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:51.260065366 +0000 UTC m=+2.827872191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:50.260161 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:50.260105    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:51.260089346 +0000 UTC m=+2.827896169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:50.318841 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:50.318808    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6492"]
Nov 01 06:09:50.322294 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:50.322256    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc618c3d8_5fa0_4e2e_a222_81b672c9b0ae.slice/crio-4a493be1126ec20bb668f49c03cb0c779916856fa2bb9225b6cbacc2a9f2289b WatchSource:0}: Error finding container 4a493be1126ec20bb668f49c03cb0c779916856fa2bb9225b6cbacc2a9f2289b: Status 404 returned error can't find the container with id 4a493be1126ec20bb668f49c03cb0c779916856fa2bb9225b6cbacc2a9f2289b
Nov 01 06:09:50.324495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:50.324459    4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:09:50.621634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:50.621592    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6492" event={"ID":"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae","Type":"ContainerStarted","Data":"4a493be1126ec20bb668f49c03cb0c779916856fa2bb9225b6cbacc2a9f2289b"}
Nov 01 06:09:51.264941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:51.264898    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:51.264941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:51.264944    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:51.265118 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:51.265059    4468 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:51.265174 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:51.265134    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.265118973 +0000 UTC m=+4.832925788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:51.265174 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:51.265059    4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:51.265239 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:51.265200    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.265188084 +0000 UTC m=+4.832994895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:51.679641 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:51.679599    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:09:52.628767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:52.628733    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6492" event={"ID":"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae","Type":"ContainerStarted","Data":"f8f76c8fa0b1c3e88da0db2b5eab529db344eb15d13ded0c7e0e6af2ccfee887"}
Nov 01 06:09:52.628963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:52.628782    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6492" event={"ID":"c618c3d8-5fa0-4e2e-a222-81b672c9b0ae","Type":"ContainerStarted","Data":"f0f7f403555a683c19df3fef6fdb3d1b9e02e38d8f4d7c0ba3d8b859b0a57284"}
Nov 01 06:09:53.279336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:53.279277    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:53.279336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:53.279336    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:53.279807 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:53.279426    4468 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:53.279807 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:53.279430    4468 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 06:09:53.279807 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:53.279500    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls podName:c5f49952-bd5a-49f7-9b11-79d3cfbceeff nodeName:}" failed. No retries permitted until 2026-11-01 06:09:57.279486277 +0000 UTC m=+8.847293087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8k87" (UID: "c5f49952-bd5a-49f7-9b11-79d3cfbceeff") : secret "insights-runtime-extractor-tls" not found
Nov 01 06:09:53.279807 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:09:53.279512    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls podName:82c5066e-78ec-49ff-ba9f-bec5967833b9 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:57.279506795 +0000 UTC m=+8.847313603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls") pod "dns-default-99jhh" (UID: "82c5066e-78ec-49ff-ba9f-bec5967833b9") : secret "dns-default-metrics-tls" not found
Nov 01 06:09:54.370037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:54.370005    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:55.168395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:55.168355    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:09:55.168596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:55.168573    4468 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 06:09:56.629821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:56.629791    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w6l2n"
Nov 01 06:09:57.304248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.304210    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:57.304248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.304247    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:57.306804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.306775    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82c5066e-78ec-49ff-ba9f-bec5967833b9-metrics-tls\") pod \"dns-default-99jhh\" (UID: \"82c5066e-78ec-49ff-ba9f-bec5967833b9\") " pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:57.306804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.306798    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5f49952-bd5a-49f7-9b11-79d3cfbceeff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8k87\" (UID: \"c5f49952-bd5a-49f7-9b11-79d3cfbceeff\") " pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:57.440042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.440007    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-99jhh"
Nov 01 06:09:57.441072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.441054    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8k87"
Nov 01 06:09:57.816435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.816405    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q8k87"]
Nov 01 06:09:57.859107 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:57.859082    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f49952_bd5a_49f7_9b11_79d3cfbceeff.slice/crio-84909e344f261be464acde28e48054dddf60d8755f1de820e5cfe63c68704140 WatchSource:0}: Error finding container 84909e344f261be464acde28e48054dddf60d8755f1de820e5cfe63c68704140: Status 404 returned error can't find the container with id 84909e344f261be464acde28e48054dddf60d8755f1de820e5cfe63c68704140
Nov 01 06:09:57.866505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:57.866460    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-99jhh"]
Nov 01 06:09:57.869946 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:09:57.869925    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c5066e_78ec_49ff_ba9f_bec5967833b9.slice/crio-ecb9830676c7b0233e58b469a443c30b52174fc7cdc81e73d91cf36e74a75e9c WatchSource:0}: Error finding container ecb9830676c7b0233e58b469a443c30b52174fc7cdc81e73d91cf36e74a75e9c: Status 404 returned error can't find the container with id ecb9830676c7b0233e58b469a443c30b52174fc7cdc81e73d91cf36e74a75e9c
Nov 01 06:09:58.644286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:58.644236    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-99jhh" event={"ID":"82c5066e-78ec-49ff-ba9f-bec5967833b9","Type":"ContainerStarted","Data":"ecb9830676c7b0233e58b469a443c30b52174fc7cdc81e73d91cf36e74a75e9c"}
Nov 01 06:09:58.645673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:58.645654    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8k87" event={"ID":"c5f49952-bd5a-49f7-9b11-79d3cfbceeff","Type":"ContainerStarted","Data":"53b1e7eba6910d6c63660a5e82dc1d96ad68d715469215fb2a04e7b40341d1b4"}
Nov 01 06:09:58.645777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:58.645691    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8k87" event={"ID":"c5f49952-bd5a-49f7-9b11-79d3cfbceeff","Type":"ContainerStarted","Data":"84909e344f261be464acde28e48054dddf60d8755f1de820e5cfe63c68704140"}
Nov 01 06:09:59.650402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:09:59.650367    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8k87" event={"ID":"c5f49952-bd5a-49f7-9b11-79d3cfbceeff","Type":"ContainerStarted","Data":"59af1cf9c189c02303489830101e0b8c1167573d34941f803c01972cfb8dbed8"}
Nov 01 06:10:00.655865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:00.655827    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-99jhh" event={"ID":"82c5066e-78ec-49ff-ba9f-bec5967833b9","Type":"ContainerStarted","Data":"a883376f661a3b7f7e8e304e30103f8111d393a2326bbec60b2264ad3e0bb224"}
Nov 01 06:10:00.656362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:00.655890    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-99jhh" event={"ID":"82c5066e-78ec-49ff-ba9f-bec5967833b9","Type":"ContainerStarted","Data":"7769618cf97cedb3ffe6dc6ffd1b7237b382967a13334268a9fb5d0dad0e09d3"}
Nov 01 06:10:00.656362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:00.655941    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-99jhh"
Nov 01 06:10:01.660985 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:01.660944    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8k87" event={"ID":"c5f49952-bd5a-49f7-9b11-79d3cfbceeff","Type":"ContainerStarted","Data":"6bcd0ab0fa0df6980c0389acdeaa1d3aa58a721aff1663be08d25e8e785a132c"}
Nov 01 06:10:02.945643 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:10:02.945610    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:10:10.663207 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.663170    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-99jhh"
Nov 01 06:10:10.847138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.847103    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"]
Nov 01 06:10:10.850192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.850172    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.891136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.891097    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbcf8d2-91c8-4732-a714-7ada04cbda4b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.891136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.891140    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afbcf8d2-91c8-4732-a714-7ada04cbda4b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.891329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.891173    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afbcf8d2-91c8-4732-a714-7ada04cbda4b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.891329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.891243    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpmt\" (UniqueName: \"kubernetes.io/projected/afbcf8d2-91c8-4732-a714-7ada04cbda4b-kube-api-access-ljpmt\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.904208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.904185    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 06:10:10.904925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.904904    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 06:10:10.904925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.904919    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 06:10:10.905092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.904963    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 06:10:10.905144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.905123    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 06:10:10.905197 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.905173    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cnkmz\""
Nov 01 06:10:10.954662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.954585    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"]
Nov 01 06:10:10.965754 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.965724    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"]
Nov 01 06:10:10.967987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.967972    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.989878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.989853    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8fqht"]
Nov 01 06:10:10.991665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991648    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afbcf8d2-91c8-4732-a714-7ada04cbda4b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.991724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991674    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpmt\" (UniqueName: \"kubernetes.io/projected/afbcf8d2-91c8-4732-a714-7ada04cbda4b-kube-api-access-ljpmt\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.991724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991694    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.991822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991724    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a184847-b655-4c64-9c4a-4f10d6a50666-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.991822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991743    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbcf8d2-91c8-4732-a714-7ada04cbda4b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.991921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991850    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:10.991921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991884    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afbcf8d2-91c8-4732-a714-7ada04cbda4b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.992026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.991951    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.992079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.992051    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.992133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.992085    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4ms\" (UniqueName: \"kubernetes.io/projected/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-api-access-tz4ms\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.992133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.992122    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a184847-b655-4c64-9c4a-4f10d6a50666-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:10.992554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.992524    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afbcf8d2-91c8-4732-a714-7ada04cbda4b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.994389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.994368    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afbcf8d2-91c8-4732-a714-7ada04cbda4b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:10.994513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:10.994460    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afbcf8d2-91c8-4732-a714-7ada04cbda4b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:11.020130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.020099    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 06:10:11.020130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.020097    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 06:10:11.020328 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.020221    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 06:10:11.020328 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.020227    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v2kcb\""
Nov 01 06:10:11.027894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.027867    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"]
Nov 01 06:10:11.048514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.048482    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpmt\" (UniqueName: \"kubernetes.io/projected/afbcf8d2-91c8-4732-a714-7ada04cbda4b-kube-api-access-ljpmt\") pod \"openshift-state-metrics-7bd7489957-bgzgv\" (UID: \"afbcf8d2-91c8-4732-a714-7ada04cbda4b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:11.093190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093154    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093192    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4ms\" (UniqueName: \"kubernetes.io/projected/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-api-access-tz4ms\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093222    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a184847-b655-4c64-9c4a-4f10d6a50666-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093248    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-root\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093278    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093306    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a184847-b655-4c64-9c4a-4f10d6a50666-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093511    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/4b9debf7-11c6-49c6-b89c-a9116021cee6-kube-api-access-wml27\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093549    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093575    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-tls\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093643    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-wtmp\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093761    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b9debf7-11c6-49c6-b89c-a9116021cee6-metrics-client-ca\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093782    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093787    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a184847-b655-4c64-9c4a-4f10d6a50666-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.093900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093808    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-textfile\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.093900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093870    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-sys\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.094302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093906    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.094302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.093993    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a184847-b655-4c64-9c4a-4f10d6a50666-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.094302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.094169    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.095936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.095914    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.096127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.096110    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.099299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.099286    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 06:10:11.120813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.120793    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 06:10:11.120813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.120796    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 06:10:11.121000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.120832    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qh54x\""
Nov 01 06:10:11.150178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.150139    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4ms\" (UniqueName: \"kubernetes.io/projected/2a184847-b655-4c64-9c4a-4f10d6a50666-kube-api-access-tz4ms\") pod \"kube-state-metrics-5dc8d7d89b-2rr5p\" (UID: \"2a184847-b655-4c64-9c4a-4f10d6a50666\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.162846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.162813    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"
Nov 01 06:10:11.194733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194697    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b9debf7-11c6-49c6-b89c-a9116021cee6-metrics-client-ca\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.194733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194742    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-textfile\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194772    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-sys\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194799    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194852    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-root\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194883    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194920    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/4b9debf7-11c6-49c6-b89c-a9116021cee6-kube-api-access-wml27\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194952    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-tls\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.194977    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-wtmp\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.195383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.195237    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-textfile\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.196659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.195991    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-sys\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.196659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.196143    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-wtmp\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.196947 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.196896    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b9debf7-11c6-49c6-b89c-a9116021cee6-root\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.197164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.197141    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.197209 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.197194    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b9debf7-11c6-49c6-b89c-a9116021cee6-metrics-client-ca\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.197849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.197826    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.198729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.198713    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b9debf7-11c6-49c6-b89c-a9116021cee6-node-exporter-tls\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.252856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.252824    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/4b9debf7-11c6-49c6-b89c-a9116021cee6-kube-api-access-wml27\") pod \"node-exporter-8fqht\" (UID: \"4b9debf7-11c6-49c6-b89c-a9116021cee6\") " pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.279605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.279573    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"
Nov 01 06:10:11.309558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.309520    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fqht"
Nov 01 06:10:11.319529 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:11.319489    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9debf7_11c6_49c6_b89c_a9116021cee6.slice/crio-f2ba2a7defed0dbda194a0db2fa3baca2c5186856927b9eab6a6f05b29791943 WatchSource:0}: Error finding container f2ba2a7defed0dbda194a0db2fa3baca2c5186856927b9eab6a6f05b29791943: Status 404 returned error can't find the container with id f2ba2a7defed0dbda194a0db2fa3baca2c5186856927b9eab6a6f05b29791943
Nov 01 06:10:11.576310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.576164    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv"]
Nov 01 06:10:11.578723 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:11.578696    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbcf8d2_91c8_4732_a714_7ada04cbda4b.slice/crio-c973b4c8eca1c6975c5770b5f14f19f9fcf05cfdb692a834ea757b870984e6a5 WatchSource:0}: Error finding container c973b4c8eca1c6975c5770b5f14f19f9fcf05cfdb692a834ea757b870984e6a5: Status 404 returned error can't find the container with id c973b4c8eca1c6975c5770b5f14f19f9fcf05cfdb692a834ea757b870984e6a5
Nov 01 06:10:11.677378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.677351    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p"]
Nov 01 06:10:11.690389 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:11.690359    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a184847_b655_4c64_9c4a_4f10d6a50666.slice/crio-2dc8aa6ad05df969480347ed53f21f47de21b5bade061465a410dd1fcfa0c07a WatchSource:0}: Error finding container 2dc8aa6ad05df969480347ed53f21f47de21b5bade061465a410dd1fcfa0c07a: Status 404 returned error can't find the container with id 2dc8aa6ad05df969480347ed53f21f47de21b5bade061465a410dd1fcfa0c07a
Nov 01 06:10:11.691332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.691268    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fqht" event={"ID":"4b9debf7-11c6-49c6-b89c-a9116021cee6","Type":"ContainerStarted","Data":"f2ba2a7defed0dbda194a0db2fa3baca2c5186856927b9eab6a6f05b29791943"}
Nov 01 06:10:11.692775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.692747    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv" event={"ID":"afbcf8d2-91c8-4732-a714-7ada04cbda4b","Type":"ContainerStarted","Data":"e4aaedf3da3836e43c89bb8a0f3369fb174af307eec00f789d506935d335e35e"}
Nov 01 06:10:11.692848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:11.692815    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv" event={"ID":"afbcf8d2-91c8-4732-a714-7ada04cbda4b","Type":"ContainerStarted","Data":"c973b4c8eca1c6975c5770b5f14f19f9fcf05cfdb692a834ea757b870984e6a5"}
Nov 01 06:10:12.446955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.446911    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8dc547477-npcfk"]
Nov 01 06:10:12.449911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.449882    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.502389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502354    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 06:10:12.502389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502358    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 06:10:12.502682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502597    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 06:10:12.502682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502645    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 06:10:12.502798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502718    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 06:10:12.502798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502654    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 06:10:12.502798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502740    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d26gh\""
Nov 01 06:10:12.502944 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.502855    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 06:10:12.504099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504073    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.504302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504108    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.504302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504133    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.504302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504218    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.504498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504321    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.504498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504374    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hk52\" (UniqueName: \"kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.504498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.504404    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.507253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.507231    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dc547477-npcfk"]
Nov 01 06:10:12.605141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605104    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605153    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605179    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605206    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605232    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605279    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hk52\" (UniqueName: \"kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605305    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.605711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.605694    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.606051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.606026    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.606155 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.606049    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.606334 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.606301    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.608279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.608258    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.608711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.608687    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.662776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.662741    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hk52\" (UniqueName: \"kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52\") pod \"console-8dc547477-npcfk\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") " pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:12.698461 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.698377    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p" event={"ID":"2a184847-b655-4c64-9c4a-4f10d6a50666","Type":"ContainerStarted","Data":"2dc8aa6ad05df969480347ed53f21f47de21b5bade061465a410dd1fcfa0c07a"}
Nov 01 06:10:12.701202 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.701165    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv" event={"ID":"afbcf8d2-91c8-4732-a714-7ada04cbda4b","Type":"ContainerStarted","Data":"2b4a86054f9755119e197e7915765ffcb70f9393f9ccec5dcfb18a3cf14eca2b"}
Nov 01 06:10:12.761573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:12.761533    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:13.177969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.177693    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dc547477-npcfk"]
Nov 01 06:10:13.208520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.208494    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:10:13.235382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.234057    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.286977 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.286944    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 06:10:13.286977 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.286952    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 06:10:13.287182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.287031    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 06:10:13.287232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.287215    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 06:10:13.287293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.287276    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-598wc\""
Nov 01 06:10:13.287335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.287281    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 06:10:13.287335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.287214    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 06:10:13.293543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.293503    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 06:10:13.293543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.293517    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 06:10:13.310250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310217    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310268    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310296    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310323    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310353    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310381    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310411    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310439    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xlf\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310507    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310540    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310591    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310619    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.310748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.310664    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.320075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.320009    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:10:13.387049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.386965    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 06:10:13.411460 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411411    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411494    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411533    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411565    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411597    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411624    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411650    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411678    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411704    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411729    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411754    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xlf\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411784    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.411905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.411810    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.412688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.412660    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.413407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.413379    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.415889 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.415865    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.417638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.416845    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.417638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.417583    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.418856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.418660    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.418856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.418816    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.418856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.418822    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.418856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.418839    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.420834 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.420807    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.423216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.423196    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.427109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.427064    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.471164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.471123    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xlf\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf\") pod \"alertmanager-main-0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.664002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.663934    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:10:13.706099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.706064    4468 generic.go:356] "Generic (PLEG): container finished" podID="4b9debf7-11c6-49c6-b89c-a9116021cee6" containerID="fa30eba4b692df5dfbfde94e7047cf2892e4cc342b07e3f67bcd5e7b70f5b313" exitCode=0
Nov 01 06:10:13.706548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.706154    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fqht" event={"ID":"4b9debf7-11c6-49c6-b89c-a9116021cee6","Type":"ContainerDied","Data":"fa30eba4b692df5dfbfde94e7047cf2892e4cc342b07e3f67bcd5e7b70f5b313"}
Nov 01 06:10:13.707410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:13.707392    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc547477-npcfk" event={"ID":"61b5a8a3-598b-4707-bfb4-684acf4dcaad","Type":"ContainerStarted","Data":"2a87f6f62658ffb1b3aeee0fec8c5b1e4de83f952ae2ac8a3626da5a8c7e41a6"}
Nov 01 06:10:14.383603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.383576    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:10:14.385912 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:14.385885    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36de36d7_194b_4c15_b1c0_48ac5bba16a0.slice/crio-9fedf620158db0232e64195be196c872871949e435ed4144760605dc893ad387 WatchSource:0}: Error finding container 9fedf620158db0232e64195be196c872871949e435ed4144760605dc893ad387: Status 404 returned error can't find the container with id 9fedf620158db0232e64195be196c872871949e435ed4144760605dc893ad387
Nov 01 06:10:14.714672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.714633    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p" event={"ID":"2a184847-b655-4c64-9c4a-4f10d6a50666","Type":"ContainerStarted","Data":"bbd03ca6c6a4ecb79259486982e45cf6bc5c09cb9876bba5a4dac3e526a723d7"}
Nov 01 06:10:14.715136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.714710    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p" event={"ID":"2a184847-b655-4c64-9c4a-4f10d6a50666","Type":"ContainerStarted","Data":"fce1143bbd1b44de08f269454c42d6a3790462583436678e04757b0a8e22c2d0"}
Nov 01 06:10:14.715136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.714731    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p" event={"ID":"2a184847-b655-4c64-9c4a-4f10d6a50666","Type":"ContainerStarted","Data":"2931f8af9da255d850d5cd5f3dec14ae371362d656fe206512d77d513f12ab2f"}
Nov 01 06:10:14.716160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.716136    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"9fedf620158db0232e64195be196c872871949e435ed4144760605dc893ad387"}
Nov 01 06:10:14.719038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.719009    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fqht" event={"ID":"4b9debf7-11c6-49c6-b89c-a9116021cee6","Type":"ContainerStarted","Data":"e907ab32370b778d29994a948634f88825946c61828382626c05e95d4609e8a6"}
Nov 01 06:10:14.719165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.719052    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fqht" event={"ID":"4b9debf7-11c6-49c6-b89c-a9116021cee6","Type":"ContainerStarted","Data":"bd5b671b34f98a2ef529bce7dd4d35dda31c41947933ebf90ba77621bf6d57f0"}
Nov 01 06:10:14.721518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.721440    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv" event={"ID":"afbcf8d2-91c8-4732-a714-7ada04cbda4b","Type":"ContainerStarted","Data":"135a95317d3ee5459019c08545a690dd0a827017d16c6a3a5dc84810ebd22a53"}
Nov 01 06:10:14.830014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.829945    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2rr5p" podStartSLOduration=2.604504485 podStartE2EDuration="4.829926625s" totalImagesPullingTime="2.22542214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:14.829132892 +0000 UTC m=+26.396939747" watchObservedRunningTime="2026-11-01 06:10:14.829926625 +0000 UTC m=+26.397733453"
Nov 01 06:10:14.942395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:14.942331    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bgzgv" podStartSLOduration=2.7328749070000002 podStartE2EDuration="4.942313482s" totalImagesPullingTime="2.209438575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:14.941689671 +0000 UTC m=+26.509496502" watchObservedRunningTime="2026-11-01 06:10:14.942313482 +0000 UTC m=+26.510120313"
Nov 01 06:10:15.056635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.056578    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8fqht" podStartSLOduration=3.142174285 podStartE2EDuration="5.056561192s" totalImagesPullingTime="1.81632302s" totalInitContainerRuntime="98.063887ms" isStatefulPod=true podCreationTimestamp="2026-11-01 06:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:15.056026183 +0000 UTC m=+26.623833035" watchObservedRunningTime="2026-11-01 06:10:15.056561192 +0000 UTC m=+26.624368021"
Nov 01 06:10:15.700869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.700829    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"]
Nov 01 06:10:15.703147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.703125    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:15.733282 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.733192    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5ld\" (UniqueName: \"kubernetes.io/projected/0ef38052-fddb-42b0-9740-f6b81a8a50e0-kube-api-access-xt5ld\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:15.733282 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.733259    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ef38052-fddb-42b0-9740-f6b81a8a50e0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:15.741642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.741611    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7687d8f685-7fxmj"]
Nov 01 06:10:15.744055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.744030    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.756159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.756134    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 06:10:15.759524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.756753    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bjk9b\""
Nov 01 06:10:15.761265 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.761235    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"]
Nov 01 06:10:15.796222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.796166    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 06:10:15.802955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.802916    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7687d8f685-7fxmj"]
Nov 01 06:10:15.803122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.803060    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 06:10:15.834676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.834638    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/88f65491-8bba-42dc-892d-3050d6e52db1-audit-log\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.834886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.834867    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2rm\" (UniqueName: \"kubernetes.io/projected/88f65491-8bba-42dc-892d-3050d6e52db1-kube-api-access-jv2rm\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.834956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.834903    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88f65491-8bba-42dc-892d-3050d6e52db1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.835011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.834951    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-secret-metrics-server-client-certs\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.835255 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.835234    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5ld\" (UniqueName: \"kubernetes.io/projected/0ef38052-fddb-42b0-9740-f6b81a8a50e0-kube-api-access-xt5ld\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:15.835290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.835276    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/88f65491-8bba-42dc-892d-3050d6e52db1-metrics-server-audit-profiles\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.835394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.835379    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-client-ca-bundle\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.835504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.835488    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ef38052-fddb-42b0-9740-f6b81a8a50e0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:15.835543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.835517    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-secret-metrics-server-tls\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.836014 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:10:15.835996    4468 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Nov 01 06:10:15.836143 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:10:15.836133    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef38052-fddb-42b0-9740-f6b81a8a50e0-monitoring-plugin-cert podName:0ef38052-fddb-42b0-9740-f6b81a8a50e0 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:16.336116489 +0000 UTC m=+27.903923297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0ef38052-fddb-42b0-9740-f6b81a8a50e0-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-4ql2v" (UID: "0ef38052-fddb-42b0-9740-f6b81a8a50e0") : secret "monitoring-plugin-cert" not found
Nov 01 06:10:15.856743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.856694    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 06:10:15.856934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.856782    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 06:10:15.856934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.856840    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-crh6m\""
Nov 01 06:10:15.856934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.856929    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e0csvo5a7m6pd\""
Nov 01 06:10:15.893783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.893748    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5ld\" (UniqueName: \"kubernetes.io/projected/0ef38052-fddb-42b0-9740-f6b81a8a50e0-kube-api-access-xt5ld\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:15.935999 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.935958    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-client-ca-bundle\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.936198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.936032    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-secret-metrics-server-tls\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.936198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.936071    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/88f65491-8bba-42dc-892d-3050d6e52db1-audit-log\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.936198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.936108    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2rm\" (UniqueName: \"kubernetes.io/projected/88f65491-8bba-42dc-892d-3050d6e52db1-kube-api-access-jv2rm\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.936198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.936131    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88f65491-8bba-42dc-892d-3050d6e52db1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.936198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.936162    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-secret-metrics-server-client-certs\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.936536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.936207    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/88f65491-8bba-42dc-892d-3050d6e52db1-metrics-server-audit-profiles\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.937095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.937065    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/88f65491-8bba-42dc-892d-3050d6e52db1-audit-log\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.937095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.937083    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88f65491-8bba-42dc-892d-3050d6e52db1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.937432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.937288    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/88f65491-8bba-42dc-892d-3050d6e52db1-metrics-server-audit-profiles\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.939623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.939599    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-client-ca-bundle\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.939740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.939725    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-secret-metrics-server-client-certs\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.956554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.956487    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/88f65491-8bba-42dc-892d-3050d6e52db1-secret-metrics-server-tls\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:15.993899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:15.993855    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2rm\" (UniqueName: \"kubernetes.io/projected/88f65491-8bba-42dc-892d-3050d6e52db1-kube-api-access-jv2rm\") pod \"metrics-server-7687d8f685-7fxmj\" (UID: \"88f65491-8bba-42dc-892d-3050d6e52db1\") " pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:16.058711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.058384    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:16.339318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.339286    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ef38052-fddb-42b0-9740-f6b81a8a50e0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:16.342033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.342001    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ef38052-fddb-42b0-9740-f6b81a8a50e0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4ql2v\" (UID: \"0ef38052-fddb-42b0-9740-f6b81a8a50e0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:16.486101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.486065    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7687d8f685-7fxmj"]
Nov 01 06:10:16.489262 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:16.489225    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f65491_8bba_42dc_892d_3050d6e52db1.slice/crio-72a87a0072105b3e7e6789c3219640df6e44594cfab4d9a87ba1377102bf8a5b WatchSource:0}: Error finding container 72a87a0072105b3e7e6789c3219640df6e44594cfab4d9a87ba1377102bf8a5b: Status 404 returned error can't find the container with id 72a87a0072105b3e7e6789c3219640df6e44594cfab4d9a87ba1377102bf8a5b
Nov 01 06:10:16.615954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.615651    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:16.625356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.625313    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-647855668f-9965p"]
Nov 01 06:10:16.628084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.628064    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.687494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.684771    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 06:10:16.687494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.684816    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fhdf8\""
Nov 01 06:10:16.687494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.684895    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 06:10:16.687494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.684974    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 06:10:16.687494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.685016    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 06:10:16.687494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.685052    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 06:10:16.735020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.734991    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-647855668f-9965p"]
Nov 01 06:10:16.735547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.735271    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 06:10:16.736362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.736333    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835" exitCode=0
Nov 01 06:10:16.736502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.736417    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835"}
Nov 01 06:10:16.737996 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.737967    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj" event={"ID":"88f65491-8bba-42dc-892d-3050d6e52db1","Type":"ContainerStarted","Data":"72a87a0072105b3e7e6789c3219640df6e44594cfab4d9a87ba1377102bf8a5b"}
Nov 01 06:10:16.741885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.741856    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-serving-certs-ca-bundle\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.741994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.741920    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-telemeter-client-tls\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.742047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.741990    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-secret-telemeter-client\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.742102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.742080    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-federate-client-tls\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.742205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.742116    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-metrics-client-ca\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.742294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.742219    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwfs\" (UniqueName: \"kubernetes.io/projected/0c6450cc-d4a7-48fb-ad27-93d76897bc71-kube-api-access-btwfs\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.742394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.742296    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.742394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.742334    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.842788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.842749    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-telemeter-client-tls\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.842968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.842795    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-secret-telemeter-client\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.842968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.842831    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-federate-client-tls\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.842968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.842867    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-metrics-client-ca\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.843179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.843128    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwfs\" (UniqueName: \"kubernetes.io/projected/0c6450cc-d4a7-48fb-ad27-93d76897bc71-kube-api-access-btwfs\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.843250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.843203    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.843307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.843259    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.844486 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.843356    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-serving-certs-ca-bundle\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.844486 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.843903    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-metrics-client-ca\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.844486 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.844090    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.844486 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.844426    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c6450cc-d4a7-48fb-ad27-93d76897bc71-serving-certs-ca-bundle\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.846070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.846043    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-telemeter-client-tls\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.846228 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.846206    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-federate-client-tls\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.846299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.846202    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-secret-telemeter-client\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.846400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.846381    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c6450cc-d4a7-48fb-ad27-93d76897bc71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.900085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.900055    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwfs\" (UniqueName: \"kubernetes.io/projected/0c6450cc-d4a7-48fb-ad27-93d76897bc71-kube-api-access-btwfs\") pod \"telemeter-client-647855668f-9965p\" (UID: \"0c6450cc-d4a7-48fb-ad27-93d76897bc71\") " pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.944004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.943969    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-647855668f-9965p"
Nov 01 06:10:16.990974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:16.990908    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"]
Nov 01 06:10:16.996811 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:16.996776    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef38052_fddb_42b0_9740_f6b81a8a50e0.slice/crio-e2f54ee9d94477c1a973ae4fdd804850c18f7cc5b608e13c9b2f4f30a332ef90 WatchSource:0}: Error finding container e2f54ee9d94477c1a973ae4fdd804850c18f7cc5b608e13c9b2f4f30a332ef90: Status 404 returned error can't find the container with id e2f54ee9d94477c1a973ae4fdd804850c18f7cc5b608e13c9b2f4f30a332ef90
Nov 01 06:10:17.367052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:17.367014    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-647855668f-9965p"]
Nov 01 06:10:17.373007 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:17.372981    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6450cc_d4a7_48fb_ad27_93d76897bc71.slice/crio-35510a73f8ebae28321d00f59268d3214cefba13b08351c63a72b4face304337 WatchSource:0}: Error finding container 35510a73f8ebae28321d00f59268d3214cefba13b08351c63a72b4face304337: Status 404 returned error can't find the container with id 35510a73f8ebae28321d00f59268d3214cefba13b08351c63a72b4face304337
Nov 01 06:10:17.744218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:17.744123    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647855668f-9965p" event={"ID":"0c6450cc-d4a7-48fb-ad27-93d76897bc71","Type":"ContainerStarted","Data":"35510a73f8ebae28321d00f59268d3214cefba13b08351c63a72b4face304337"}
Nov 01 06:10:17.745929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:17.745899    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc547477-npcfk" event={"ID":"61b5a8a3-598b-4707-bfb4-684acf4dcaad","Type":"ContainerStarted","Data":"08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65"}
Nov 01 06:10:17.747393 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:17.747368    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v" event={"ID":"0ef38052-fddb-42b0-9740-f6b81a8a50e0","Type":"ContainerStarted","Data":"e2f54ee9d94477c1a973ae4fdd804850c18f7cc5b608e13c9b2f4f30a332ef90"}
Nov 01 06:10:17.859513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:17.859438    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8dc547477-npcfk" podStartSLOduration=2.195265482 podStartE2EDuration="5.859422732s" totalImagesPullingTime="3.66415725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:17.859012509 +0000 UTC m=+29.426819339" watchObservedRunningTime="2026-11-01 06:10:17.859422732 +0000 UTC m=+29.427229560"
Nov 01 06:10:18.357222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:18.355780    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rnnc"
Nov 01 06:10:18.755391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:18.755313    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj" event={"ID":"88f65491-8bba-42dc-892d-3050d6e52db1","Type":"ContainerStarted","Data":"004e9ea73b372cd7069d4521a158209768f6285ec872e0396a627e923825dcdb"}
Nov 01 06:10:18.875398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:18.871571    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj" podStartSLOduration=1.861307773 podStartE2EDuration="3.871554772s" totalImagesPullingTime="2.010246999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:18.870119008 +0000 UTC m=+30.437925851" watchObservedRunningTime="2026-11-01 06:10:18.871554772 +0000 UTC m=+30.439361602"
Nov 01 06:10:19.761016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.760988    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v" event={"ID":"0ef38052-fddb-42b0-9740-f6b81a8a50e0","Type":"ContainerStarted","Data":"73441c590123095083deabbbab8f28fd5b3121e44b91c1785e109ffbc63914c7"}
Nov 01 06:10:19.761512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.761101    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:19.765775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.765745    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308"}
Nov 01 06:10:19.765962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.765827    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267"}
Nov 01 06:10:19.765962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.765869    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc"}
Nov 01 06:10:19.765962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.765906    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13"}
Nov 01 06:10:19.765962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.765947    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e"}
Nov 01 06:10:19.767231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.767210    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v"
Nov 01 06:10:19.768193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.768172    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647855668f-9965p" event={"ID":"0c6450cc-d4a7-48fb-ad27-93d76897bc71","Type":"ContainerStarted","Data":"e53570c186a7634f3484c670ef0e233f4cca840a4993ad1037df4558789b4548"}
Nov 01 06:10:19.958776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.958677    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4ql2v" podStartSLOduration=2.93369067 podStartE2EDuration="4.958658501s" totalImagesPullingTime="2.024967831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:19.875228454 +0000 UTC m=+31.443035285" watchObservedRunningTime="2026-11-01 06:10:19.958658501 +0000 UTC m=+31.526465340"
Nov 01 06:10:19.958930 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.958828    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:10:19.980677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:19.980645    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.025705 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.025673    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:10:20.033133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033101    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 06:10:20.033133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033104    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 06:10:20.033318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033167    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 06:10:20.033318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033239    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 06:10:20.033870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033784    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 06:10:20.033870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033796    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 06:10:20.033870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033801    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 06:10:20.033870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033803    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 06:10:20.033870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033824    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 06:10:20.033870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.033839    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 06:10:20.072017 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.071980    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072017 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072022    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072043    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072068    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072151    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072213    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072244    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072265    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072300    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072324    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072341    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072423    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072526    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072575    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072714 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072673    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072716    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072740    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072780    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072820    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.072917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.072846    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlnjq\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.073867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.073826    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-74g68\""
Nov 01 06:10:20.073867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.073831    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b96qfv7d18d41\""
Nov 01 06:10:20.074017 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.073932    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 06:10:20.074273 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.074247    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 06:10:20.124379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.124343    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 06:10:20.124379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.124354    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 06:10:20.173779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.173741    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.173779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.173786    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.173812    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.173971    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174013    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174044    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnjq\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174092    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174144    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174173    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174219    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174260    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174287    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174323    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174352    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174377    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174405    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174435    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174488    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174528    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174585    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.174905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.174814    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.176996 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.175570    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.177826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.177793    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.178658 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.178580    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.180761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.178829    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.180761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.179221    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.180761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.180230    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.180761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.180592    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.181139 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.181118    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.181774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.181688    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.181871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.181849    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.181938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.181887    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.182372 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.182289    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.183016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.182990    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.183454 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.183432    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.186854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.186826    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.187566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.187544    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.188562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.188539    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.189366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.189346    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.232615 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.232528    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnjq\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq\") pod \"prometheus-k8s-0\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.399933 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.399901    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:20.774005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.773922    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647855668f-9965p" event={"ID":"0c6450cc-d4a7-48fb-ad27-93d76897bc71","Type":"ContainerStarted","Data":"e5b95d28fbdde3c818eefdde0e1599f6638c0cf9d41bcbfaf2826935279945f8"}
Nov 01 06:10:20.774005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.773976    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-647855668f-9965p" event={"ID":"0c6450cc-d4a7-48fb-ad27-93d76897bc71","Type":"ContainerStarted","Data":"29ea20065835785b3995d6c531a988f75ba76e8f54a9797e1f2137e359ec6c51"}
Nov 01 06:10:20.890957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.890907    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-647855668f-9965p" podStartSLOduration=2.592511039 podStartE2EDuration="4.890887803s" totalImagesPullingTime="2.298376764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:20.889772199 +0000 UTC m=+32.457579028" watchObservedRunningTime="2026-11-01 06:10:20.890887803 +0000 UTC m=+32.458694633"
Nov 01 06:10:20.914728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:20.914694    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:10:20.944529 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:20.944502    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aad22ee_69b6_4fb0_8d9c_d6d4193288fc.slice/crio-2fdba453bd62eabe6e8384fea98537ff517030bd0042986376e9888825e58d98 WatchSource:0}: Error finding container 2fdba453bd62eabe6e8384fea98537ff517030bd0042986376e9888825e58d98: Status 404 returned error can't find the container with id 2fdba453bd62eabe6e8384fea98537ff517030bd0042986376e9888825e58d98
Nov 01 06:10:21.683089 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:21.683056    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7llbh"
Nov 01 06:10:21.781195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:21.781159    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerStarted","Data":"e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d"}
Nov 01 06:10:21.782681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:21.782653    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5" exitCode=0
Nov 01 06:10:21.782793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:21.782734    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"}
Nov 01 06:10:21.782838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:21.782796    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"2fdba453bd62eabe6e8384fea98537ff517030bd0042986376e9888825e58d98"}
Nov 01 06:10:22.036559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.036445    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.641137128 podStartE2EDuration="9.036428571s" totalImagesPullingTime="5.380995516s" totalInitContainerRuntime="14.295927ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:22.034821974 +0000 UTC m=+33.602628829" watchObservedRunningTime="2026-11-01 06:10:22.036428571 +0000 UTC m=+33.604235395"
Nov 01 06:10:22.415734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.415688    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f8fb769-njhkb"]
Nov 01 06:10:22.448924 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.448890    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.476393 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.476352    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f8fb769-njhkb"]
Nov 01 06:10:22.495156 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495123    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495165    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnl4x\" (UniqueName: \"kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495187    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495261    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495411    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495484 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495440    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495498    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.495598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.495584    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.550847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.550811    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 06:10:22.596018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.595982    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596025    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnl4x\" (UniqueName: \"kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596044    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596059    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596095    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596118    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596148    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596204    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596646    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.596952 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596925    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.597046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.596925    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.597103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.597085    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.597158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.597106    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.599223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.599199    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.599296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.599279    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.652090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.652062    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnl4x\" (UniqueName: \"kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x\") pod \"console-86f8fb769-njhkb\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") " pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.761893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.761814    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:22.761893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.761873    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:22.766908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.766886    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:22.768014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.767992    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:22.792278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:22.792240    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:23.160883 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:10:23.160843    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029a78e5_774b_4d43_a8ac_cfbfa67f5c93.slice/crio-7f1b79b1a0f8c768f36f56fa50c355de45868da167148d2287d6348cace6d7f8 WatchSource:0}: Error finding container 7f1b79b1a0f8c768f36f56fa50c355de45868da167148d2287d6348cace6d7f8: Status 404 returned error can't find the container with id 7f1b79b1a0f8c768f36f56fa50c355de45868da167148d2287d6348cace6d7f8
Nov 01 06:10:23.162628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:23.162600    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f8fb769-njhkb"]
Nov 01 06:10:23.793386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:23.793345    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f8fb769-njhkb" event={"ID":"029a78e5-774b-4d43-a8ac-cfbfa67f5c93","Type":"ContainerStarted","Data":"24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093"}
Nov 01 06:10:23.793897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:23.793399    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f8fb769-njhkb" event={"ID":"029a78e5-774b-4d43-a8ac-cfbfa67f5c93","Type":"ContainerStarted","Data":"7f1b79b1a0f8c768f36f56fa50c355de45868da167148d2287d6348cace6d7f8"}
Nov 01 06:10:23.908390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:23.908330    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86f8fb769-njhkb" podStartSLOduration=1.9083161130000001 podStartE2EDuration="1.908316113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:23.907086324 +0000 UTC m=+35.474893154" watchObservedRunningTime="2026-11-01 06:10:23.908316113 +0000 UTC m=+35.476122940"
Nov 01 06:10:25.804818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:25.804785    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"}
Nov 01 06:10:25.805248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:25.804872    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"}
Nov 01 06:10:27.820213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:27.820180    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"}
Nov 01 06:10:27.820753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:27.820240    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"}
Nov 01 06:10:27.820753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:27.820267    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"}
Nov 01 06:10:27.820753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:27.820292    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerStarted","Data":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"}
Nov 01 06:10:28.094012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:28.093963    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.44148734 podStartE2EDuration="9.093947049s" totalImagesPullingTime="5.629464332s" totalInitContainerRuntime="22.995377ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:28.043694595 +0000 UTC m=+39.611501437" watchObservedRunningTime="2026-11-01 06:10:28.093947049 +0000 UTC m=+39.661753880"
Nov 01 06:10:30.401032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:30.400945    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:10:32.768503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:32.768447    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:32.768922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:32.768516    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:32.772990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:32.772967    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:32.843399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:32.843369    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:10:32.948624 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:10:32.948585    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:10:33.030107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:33.030020    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dc547477-npcfk"]
Nov 01 06:10:36.059170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:36.059131    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:36.059584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:36.059211    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:56.064754 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:56.064724    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:56.068713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:56.068689    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7687d8f685-7fxmj"
Nov 01 06:10:58.050873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.050807    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8dc547477-npcfk" podUID="61b5a8a3-598b-4707-bfb4-684acf4dcaad" containerName="console" containerID="cri-o://08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65" gracePeriod=15
Nov 01 06:10:58.288892 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.288868    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:58.316093 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316024    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316093 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316067    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316122    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316142    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316189    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316216    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52\" (UniqueName: \"kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316237    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config\") pod \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\" (UID: \"61b5a8a3-598b-4707-bfb4-684acf4dcaad\") "
Nov 01 06:10:58.316519 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316365    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:10:58.316519 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316447    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:58.316519 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316461    4468 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b5a8a3-598b-4707-bfb4-684acf4dcaad-tmp\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.316747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316721    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:58.316887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.316796    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:58.318706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.318682    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:10:58.318815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.318762    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:10:58.318815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.318784    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52" pod "61b5a8a3-598b-4707-bfb4-684acf4dcaad" (UID: "61b5a8a3-598b-4707-bfb4-684acf4dcaad"). InnerVolumeSpecName "kube-api-access-5hk52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:58.417062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.417033    4468 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-oauth-serving-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.417062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.417059    4468 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-serving-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.417062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.417068    4468 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.417280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.417077    4468 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-oauth-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.417280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.417087    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hk52\" (UniqueName: \"kubernetes.io/projected/61b5a8a3-598b-4707-bfb4-684acf4dcaad-kube-api-access-5hk52\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.417280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.417097    4468 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b5a8a3-598b-4707-bfb4-684acf4dcaad-console-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:58.943388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.943284    4468 generic.go:356] "Generic (PLEG): container finished" podID="61b5a8a3-598b-4707-bfb4-684acf4dcaad" containerID="08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65" exitCode=0
Nov 01 06:10:58.943388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.943334    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc547477-npcfk" event={"ID":"61b5a8a3-598b-4707-bfb4-684acf4dcaad","Type":"ContainerDied","Data":"08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65"}
Nov 01 06:10:58.943388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.943349    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc547477-npcfk"
Nov 01 06:10:58.943388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.943387    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc547477-npcfk" event={"ID":"61b5a8a3-598b-4707-bfb4-684acf4dcaad","Type":"ContainerDied","Data":"2a87f6f62658ffb1b3aeee0fec8c5b1e4de83f952ae2ac8a3626da5a8c7e41a6"}
Nov 01 06:10:58.943721 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.943413    4468 scope.go:122] "RemoveContainer" containerID="08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65"
Nov 01 06:10:58.951651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.951632    4468 scope.go:122] "RemoveContainer" containerID="08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65"
Nov 01 06:10:58.951954 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:10:58.951933    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65\": container with ID starting with 08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65 not found: ID does not exist" containerID="08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65"
Nov 01 06:10:58.952019 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:58.951965    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65"} err="failed to get container status \"08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65\": rpc error: code = NotFound desc = could not find container \"08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65\": container with ID starting with 08fff5973a6e7fc521f315542f51b1b5730397ce495ac4caf6d0cf2da874ae65 not found: ID does not exist"
Nov 01 06:10:59.105393 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:59.105361    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dc547477-npcfk"]
Nov 01 06:10:59.154875 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:10:59.154839    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8dc547477-npcfk"]
Nov 01 06:11:00.587548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:00.587519    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b5a8a3-598b-4707-bfb4-684acf4dcaad" path="/var/lib/kubelet/pods/61b5a8a3-598b-4707-bfb4-684acf4dcaad/volumes"
Nov 01 06:11:00.670674 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:00.670633    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:00.690124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:00.690097    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:00.968369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:00.968288    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:02.947230 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:02.947186    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:11:05.152105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.152069    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dn84m-krqpd"]
Nov 01 06:11:05.152595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.152289    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-dn84m-krqpd" podUID="cbf40282-bfe4-4279-8304-3173ec43dcf1" containerName="insights-gathering" containerID="cri-o://f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069" gracePeriod=30
Nov 01 06:11:05.402143 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.402084    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:11:05.485451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485406    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") pod \"cbf40282-bfe4-4279-8304-3173ec43dcf1\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") "
Nov 01 06:11:05.485671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485507    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") pod \"cbf40282-bfe4-4279-8304-3173ec43dcf1\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") "
Nov 01 06:11:05.485671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485602    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") pod \"cbf40282-bfe4-4279-8304-3173ec43dcf1\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") "
Nov 01 06:11:05.485671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485642    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") pod \"cbf40282-bfe4-4279-8304-3173ec43dcf1\" (UID: \"cbf40282-bfe4-4279-8304-3173ec43dcf1\") "
Nov 01 06:11:05.485854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485822    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle" pod "cbf40282-bfe4-4279-8304-3173ec43dcf1" (UID: "cbf40282-bfe4-4279-8304-3173ec43dcf1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:05.485894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485868    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path" pod "cbf40282-bfe4-4279-8304-3173ec43dcf1" (UID: "cbf40282-bfe4-4279-8304-3173ec43dcf1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:11:05.485948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.485933    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle" pod "cbf40282-bfe4-4279-8304-3173ec43dcf1" (UID: "cbf40282-bfe4-4279-8304-3173ec43dcf1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:05.487993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.487970    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh" pod "cbf40282-bfe4-4279-8304-3173ec43dcf1" (UID: "cbf40282-bfe4-4279-8304-3173ec43dcf1"). InnerVolumeSpecName "kube-api-access-xbqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:05.586243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.586200    4468 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-service-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:05.586243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.586231    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqmh\" (UniqueName: \"kubernetes.io/projected/cbf40282-bfe4-4279-8304-3173ec43dcf1-kube-api-access-xbqmh\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:05.586243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.586241    4468 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf40282-bfe4-4279-8304-3173ec43dcf1-trusted-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:05.586243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.586251    4468 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbf40282-bfe4-4279-8304-3173ec43dcf1-archives-path\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:05.969205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.969172    4468 generic.go:356] "Generic (PLEG): container finished" podID="cbf40282-bfe4-4279-8304-3173ec43dcf1" containerID="f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069" exitCode=2
Nov 01 06:11:05.969394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.969223    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dn84m-krqpd"
Nov 01 06:11:05.969394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.969252    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dn84m-krqpd" event={"ID":"cbf40282-bfe4-4279-8304-3173ec43dcf1","Type":"ContainerDied","Data":"f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069"}
Nov 01 06:11:05.969394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.969310    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dn84m-krqpd" event={"ID":"cbf40282-bfe4-4279-8304-3173ec43dcf1","Type":"ContainerDied","Data":"d3736e1e6359b6c524418e91c68f6cf5883615bbc1f932e7f23d7b3c0efc25d1"}
Nov 01 06:11:05.969394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.969337    4468 scope.go:122] "RemoveContainer" containerID="f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069"
Nov 01 06:11:05.977541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.977524    4468 scope.go:122] "RemoveContainer" containerID="f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069"
Nov 01 06:11:05.977847 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:05.977828    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069\": container with ID starting with f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069 not found: ID does not exist" containerID="f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069"
Nov 01 06:11:05.977909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:05.977861    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069"} err="failed to get container status \"f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069\": rpc error: code = NotFound desc = could not find container \"f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069\": container with ID starting with f1d649ac4d3660bc27848346a423f9589c0a84ff9353454c8562bc75175d7069 not found: ID does not exist"
Nov 01 06:11:06.134913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:06.132425    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dn84m-krqpd"]
Nov 01 06:11:06.180670 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:06.180638    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-dn84m-krqpd"]
Nov 01 06:11:06.588061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:06.588028    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf40282-bfe4-4279-8304-3173ec43dcf1" path="/var/lib/kubelet/pods/cbf40282-bfe4-4279-8304-3173ec43dcf1/volumes"
Nov 01 06:11:17.336690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.336655    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:11:17.337232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.337192    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="alertmanager" containerID="cri-o://4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e" gracePeriod=120
Nov 01 06:11:17.337384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.337228    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-metric" containerID="cri-o://d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308" gracePeriod=120
Nov 01 06:11:17.337384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.337257    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="config-reloader" containerID="cri-o://de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13" gracePeriod=120
Nov 01 06:11:17.337384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.337323    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy" containerID="cri-o://6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267" gracePeriod=120
Nov 01 06:11:17.337384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.337255    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-web" containerID="cri-o://5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc" gracePeriod=120
Nov 01 06:11:17.337626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:17.337329    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="prom-label-proxy" containerID="cri-o://e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d" gracePeriod=120
Nov 01 06:11:18.015574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015543    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d" exitCode=0
Nov 01 06:11:18.015574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015568    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308" exitCode=0
Nov 01 06:11:18.015574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015574    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267" exitCode=0
Nov 01 06:11:18.015574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015580    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13" exitCode=0
Nov 01 06:11:18.015836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015585    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e" exitCode=0
Nov 01 06:11:18.015836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015613    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d"}
Nov 01 06:11:18.015836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015676    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308"}
Nov 01 06:11:18.015836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015703    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267"}
Nov 01 06:11:18.015836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015724    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13"}
Nov 01 06:11:18.015836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.015748    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e"}
Nov 01 06:11:18.589195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.589170    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:18.705682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705588    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.705682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705641    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.705682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705666    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.705968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705796    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.705968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705861    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.705968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705897    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.705968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705950    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.705989    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706018    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706078    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706120    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706054    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:18.706390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706157    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706182    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:11:18.706390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706199    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config\") pod \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\" (UID: \"36de36d7-194b-4c15-b1c0-48ac5bba16a0\") "
Nov 01 06:11:18.706583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706566    4468 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.706649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706589    4468 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-alertmanager-main-db\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.706705 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.706656    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:18.709270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.709238    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.709566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.709540    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.709653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.709558    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.709720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.709671    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:18.709935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.709890    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:11:18.710531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.710504    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "kube-api-access-j2xlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:18.710622 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.710571    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.710780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.710762    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.715072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.715044    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.720769 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.720744    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config" pod "36de36d7-194b-4c15-b1c0-48ac5bba16a0" (UID: "36de36d7-194b-4c15-b1c0-48ac5bba16a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:18.807710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807679    4468 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-tls-assets\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807709    4468 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807720    4468 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-main-tls\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807730    4468 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36de36d7-194b-4c15-b1c0-48ac5bba16a0-metrics-client-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807740    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2xlf\" (UniqueName: \"kubernetes.io/projected/36de36d7-194b-4c15-b1c0-48ac5bba16a0-kube-api-access-j2xlf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807750    4468 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807758    4468 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807766    4468 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-cluster-tls-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807776    4468 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807786    4468 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36de36d7-194b-4c15-b1c0-48ac5bba16a0-web-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:18.807956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:18.807794    4468 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36de36d7-194b-4c15-b1c0-48ac5bba16a0-config-out\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:19.022087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.021990    4468 generic.go:356] "Generic (PLEG): container finished" podID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerID="5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc" exitCode=0
Nov 01 06:11:19.022087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.022074    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc"}
Nov 01 06:11:19.022317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.022092    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.022317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.022151    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36de36d7-194b-4c15-b1c0-48ac5bba16a0","Type":"ContainerDied","Data":"9fedf620158db0232e64195be196c872871949e435ed4144760605dc893ad387"}
Nov 01 06:11:19.022317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.022186    4468 scope.go:122] "RemoveContainer" containerID="e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d"
Nov 01 06:11:19.030756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.030739    4468 scope.go:122] "RemoveContainer" containerID="d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308"
Nov 01 06:11:19.038000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.037979    4468 scope.go:122] "RemoveContainer" containerID="6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267"
Nov 01 06:11:19.045493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.045452    4468 scope.go:122] "RemoveContainer" containerID="5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc"
Nov 01 06:11:19.052632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.052613    4468 scope.go:122] "RemoveContainer" containerID="de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13"
Nov 01 06:11:19.060173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.060149    4468 scope.go:122] "RemoveContainer" containerID="4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e"
Nov 01 06:11:19.067932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.067914    4468 scope.go:122] "RemoveContainer" containerID="2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835"
Nov 01 06:11:19.075442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.075422    4468 scope.go:122] "RemoveContainer" containerID="e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d"
Nov 01 06:11:19.075803 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.075782    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d\": container with ID starting with e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d not found: ID does not exist" containerID="e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d"
Nov 01 06:11:19.075858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.075813    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d"} err="failed to get container status \"e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d\": rpc error: code = NotFound desc = could not find container \"e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d\": container with ID starting with e33451c7f0ccff4a013ba92d412fb0b5b5d515389615cf947c8e298a9a0bd02d not found: ID does not exist"
Nov 01 06:11:19.075858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.075829    4468 scope.go:122] "RemoveContainer" containerID="d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308"
Nov 01 06:11:19.076099 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.076067    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308\": container with ID starting with d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308 not found: ID does not exist" containerID="d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308"
Nov 01 06:11:19.076165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076110    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308"} err="failed to get container status \"d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308\": rpc error: code = NotFound desc = could not find container \"d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308\": container with ID starting with d301ce422bb5c51b92af28a69222c88a5e1e46bff2a67333a825dd68952a5308 not found: ID does not exist"
Nov 01 06:11:19.076165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076136    4468 scope.go:122] "RemoveContainer" containerID="6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267"
Nov 01 06:11:19.076402 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.076386    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267\": container with ID starting with 6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267 not found: ID does not exist" containerID="6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267"
Nov 01 06:11:19.076492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076410    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267"} err="failed to get container status \"6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267\": rpc error: code = NotFound desc = could not find container \"6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267\": container with ID starting with 6e82041e286663f039315aba53f9be1c113744712f489ffd96fa80235fcc5267 not found: ID does not exist"
Nov 01 06:11:19.076492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076430    4468 scope.go:122] "RemoveContainer" containerID="5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc"
Nov 01 06:11:19.076689 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.076672    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc\": container with ID starting with 5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc not found: ID does not exist" containerID="5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc"
Nov 01 06:11:19.076734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076695    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc"} err="failed to get container status \"5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc\": rpc error: code = NotFound desc = could not find container \"5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc\": container with ID starting with 5b6e417395ca6dbe451c879aa28ebe62c63729cc4dee70c2b8efb38b06482cdc not found: ID does not exist"
Nov 01 06:11:19.076734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076709    4468 scope.go:122] "RemoveContainer" containerID="de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13"
Nov 01 06:11:19.076960 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.076940    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13\": container with ID starting with de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13 not found: ID does not exist" containerID="de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13"
Nov 01 06:11:19.077000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076967    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13"} err="failed to get container status \"de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13\": rpc error: code = NotFound desc = could not find container \"de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13\": container with ID starting with de41f283127c1a592df1a923b37069c93aa0ecd80b74ba4766c72ca9d3c37d13 not found: ID does not exist"
Nov 01 06:11:19.077000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.076982    4468 scope.go:122] "RemoveContainer" containerID="4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e"
Nov 01 06:11:19.077200 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.077181    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e\": container with ID starting with 4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e not found: ID does not exist" containerID="4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e"
Nov 01 06:11:19.077237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.077203    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e"} err="failed to get container status \"4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e\": rpc error: code = NotFound desc = could not find container \"4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e\": container with ID starting with 4ecfe69583272758bdb2684b9be3793b3880652a85801e937fcbeb110c1e590e not found: ID does not exist"
Nov 01 06:11:19.077237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.077217    4468 scope.go:122] "RemoveContainer" containerID="2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835"
Nov 01 06:11:19.077432 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:19.077416    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835\": container with ID starting with 2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835 not found: ID does not exist" containerID="2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835"
Nov 01 06:11:19.077491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.077437    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835"} err="failed to get container status \"2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835\": rpc error: code = NotFound desc = could not find container \"2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835\": container with ID starting with 2d51bf07ea5c9da41fbf8331a566b474ca0f6fd6534df5ffa71dd1604789b835 not found: ID does not exist"
Nov 01 06:11:19.188124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.188089    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:11:19.191713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.191690    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:11:19.238403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238361    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:11:19.238682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238668    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="config-reloader"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238683    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="config-reloader"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238691    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-web"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238696    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-web"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238707    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="prom-label-proxy"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238712    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="prom-label-proxy"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238719    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="init-config-reloader"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238724    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="init-config-reloader"
Nov 01 06:11:19.238728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238729    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238735    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238741    4468 cpu_manager.go:414] "containerMap: removing container" podUID="cbf40282-bfe4-4279-8304-3173ec43dcf1" containerName="insights-gathering"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238745    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf40282-bfe4-4279-8304-3173ec43dcf1" containerName="insights-gathering"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238751    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-metric"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238756    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-metric"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238768    4468 cpu_manager.go:414] "containerMap: removing container" podUID="61b5a8a3-598b-4707-bfb4-684acf4dcaad" containerName="console"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238772    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b5a8a3-598b-4707-bfb4-684acf4dcaad" containerName="console"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238783    4468 cpu_manager.go:414] "containerMap: removing container" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="alertmanager"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238788    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="alertmanager"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238830    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b5a8a3-598b-4707-bfb4-684acf4dcaad" containerName="console"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238839    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="alertmanager"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238846    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="config-reloader"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238851    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-web"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238857    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="prom-label-proxy"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238862    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238867    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf40282-bfe4-4279-8304-3173ec43dcf1" containerName="insights-gathering"
Nov 01 06:11:19.238974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.238872    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" containerName="kube-rbac-proxy-metric"
Nov 01 06:11:19.242554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.242539    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.295398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295354    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 06:11:19.295398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295379    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 06:11:19.295660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295401    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 06:11:19.295660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295442    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 06:11:19.295660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295645    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 06:11:19.295660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295640    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 06:11:19.295812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295666    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 06:11:19.295812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295692    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-598wc\""
Nov 01 06:11:19.295812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.295803    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 06:11:19.312623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312576    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312624    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgd5\" (UniqueName: \"kubernetes.io/projected/0100c589-3e24-459b-8baf-ac0337238eb6-kube-api-access-qrgd5\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312698    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312757    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0100c589-3e24-459b-8baf-ac0337238eb6-config-out\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312811    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0100c589-3e24-459b-8baf-ac0337238eb6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312851    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312892 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312869    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312916    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312938    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0100c589-3e24-459b-8baf-ac0337238eb6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.312997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.312974    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0100c589-3e24-459b-8baf-ac0337238eb6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.313048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.313037    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0100c589-3e24-459b-8baf-ac0337238eb6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.313079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.313057    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-web-config\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.313110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.313098    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-config-volume\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.345122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.345089    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 06:11:19.356980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.356953    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:11:19.415547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415502    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415554    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0100c589-3e24-459b-8baf-ac0337238eb6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415587    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0100c589-3e24-459b-8baf-ac0337238eb6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415616    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0100c589-3e24-459b-8baf-ac0337238eb6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415632    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-web-config\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415658    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-config-volume\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415684    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415774    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgd5\" (UniqueName: \"kubernetes.io/projected/0100c589-3e24-459b-8baf-ac0337238eb6-kube-api-access-qrgd5\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.415818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415817    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.416203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415852    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0100c589-3e24-459b-8baf-ac0337238eb6-config-out\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.416203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415893    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0100c589-3e24-459b-8baf-ac0337238eb6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.416203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415945    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.416203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.415969    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.416203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.416090    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0100c589-3e24-459b-8baf-ac0337238eb6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.416920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.416892    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0100c589-3e24-459b-8baf-ac0337238eb6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.418516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.418486    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0100c589-3e24-459b-8baf-ac0337238eb6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.418721    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.418797    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-config-volume\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.418934    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.419019    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.419625    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-web-config\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.419781    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0100c589-3e24-459b-8baf-ac0337238eb6-config-out\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.419993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.419968    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0100c589-3e24-459b-8baf-ac0337238eb6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.420794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.420777    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.421219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.421199    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0100c589-3e24-459b-8baf-ac0337238eb6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.474404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.474376    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgd5\" (UniqueName: \"kubernetes.io/projected/0100c589-3e24-459b-8baf-ac0337238eb6-kube-api-access-qrgd5\") pod \"alertmanager-main-0\" (UID: \"0100c589-3e24-459b-8baf-ac0337238eb6\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.561216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.561130    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 06:11:19.983922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:19.983838    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 06:11:20.027454 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:20.027422    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"10908e7c495ec9b7587cb283286df7e7c476fbaaed6be57792fba0692346d114"}
Nov 01 06:11:20.587508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:20.587457    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36de36d7-194b-4c15-b1c0-48ac5bba16a0" path="/var/lib/kubelet/pods/36de36d7-194b-4c15-b1c0-48ac5bba16a0/volumes"
Nov 01 06:11:21.033115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:21.033031    4468 generic.go:356] "Generic (PLEG): container finished" podID="0100c589-3e24-459b-8baf-ac0337238eb6" containerID="a9d036941982b1b67dbb7ca888ab4aa5e1c4fe677a84d3a63bec6d0dabd65596" exitCode=0
Nov 01 06:11:21.033115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:21.033103    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerDied","Data":"a9d036941982b1b67dbb7ca888ab4aa5e1c4fe677a84d3a63bec6d0dabd65596"}
Nov 01 06:11:22.040315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.040270    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"341598e29ee30837e3ca053c5e964ab943a43f6440320be90f4e558dd75e00b6"}
Nov 01 06:11:22.040739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.040355    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"8ff445cd888387cee3c4b778748f4ee7cb116100c3a924428913feee94f3fbc9"}
Nov 01 06:11:22.040739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.040395    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"01d210c54132e5e8cdc1907e683cd512fd70e26f17a47c6c2d143488f6ea4fe4"}
Nov 01 06:11:22.040739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.040432    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"bbb1dfc32be65815e0159bc88ce721fe98c53778e4c8ed4873c00a9d5dd2ee57"}
Nov 01 06:11:22.040739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.040491    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"d6b335bdaa258f2bd89e6c56d68b7a26eab57f947a4cb31c26e06b9947618431"}
Nov 01 06:11:22.040739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.040530    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0100c589-3e24-459b-8baf-ac0337238eb6","Type":"ContainerStarted","Data":"164e93405ba961070c9c769134ff1d89d4bb5c491721e15bf458a06c0b8469b0"}
Nov 01 06:11:22.161456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:22.161404    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.147585119 podStartE2EDuration="3.161375568s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.790449ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:22.160074776 +0000 UTC m=+93.727881631" watchObservedRunningTime="2026-11-01 06:11:22.161375568 +0000 UTC m=+93.729182398"
Nov 01 06:11:23.037272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037235    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:11:23.037730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037690    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="prometheus" containerID="cri-o://3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67" gracePeriod=600
Nov 01 06:11:23.037730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037701    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7" gracePeriod=600
Nov 01 06:11:23.037928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037716    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="config-reloader" containerID="cri-o://12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4" gracePeriod=600
Nov 01 06:11:23.037928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037691    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="thanos-sidecar" containerID="cri-o://3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c" gracePeriod=600
Nov 01 06:11:23.037928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037762    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy" containerID="cri-o://07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb" gracePeriod=600
Nov 01 06:11:23.037928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.037719    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-web" containerID="cri-o://4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e" gracePeriod=600
Nov 01 06:11:23.294127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.294066    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:23.452083 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452051    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452265 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452098    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452265 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452129    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452256    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452522    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.452626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452552    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452585    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452619    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452657    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452745    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.452789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452776    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452803    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452833    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452869    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452913    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452948    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.452987    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453028    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453061    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453093    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.453595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453140    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453172    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out\") pod \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\" (UID: \"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc\") "
Nov 01 06:11:23.453595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453172    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.453595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453438    4468 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.453595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453457    4468 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.453595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453495    4468 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.454005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.453905    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.455057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.455027    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:11:23.455580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.455552    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.455661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.455629    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "kube-api-access-xlnjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:23.455721 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.455708    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.455974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.455952    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.456319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.456282    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:11:23.456402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.456328    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.456599 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.456563    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.456761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.456735    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.457028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.457003    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:11:23.457385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.457350    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.457869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.457836    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.458628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.458601    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.458841 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.458813    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.458946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.458881    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:23.470743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.470707    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config" pod "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" (UID: "0aad22ee-69b6-4fb0-8d9c-d6d4193288fc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:23.554499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554445    4468 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-kube-rbac-proxy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554498    4468 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554508    4468 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554518    4468 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-grpc-tls\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554526    4468 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-metrics-client-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554535    4468 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554544    4468 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config-out\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554551    4468 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554560    4468 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554569    4468 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554577    4468 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554585    4468 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-tls-assets\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554594    4468 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-db\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554603    4468 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-configmap-metrics-client-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554619    4468 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-web-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554627    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlnjq\" (UniqueName: \"kubernetes.io/projected/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-kube-api-access-xlnjq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:23.554691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:23.554636    4468 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:24.059305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059277    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7" exitCode=0
Nov 01 06:11:24.059305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059300    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb" exitCode=0
Nov 01 06:11:24.059305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059308    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e" exitCode=0
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059317    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c" exitCode=0
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059325    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4" exitCode=0
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059335    4468 generic.go:356] "Generic (PLEG): container finished" podID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67" exitCode=0
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059372    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059378    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"}
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059454    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"}
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059488    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059510    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"}
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059535    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"}
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059566    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"}
Nov 01 06:11:24.059608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059591    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"}
Nov 01 06:11:24.060109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.059626    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0aad22ee-69b6-4fb0-8d9c-d6d4193288fc","Type":"ContainerDied","Data":"2fdba453bd62eabe6e8384fea98537ff517030bd0042986376e9888825e58d98"}
Nov 01 06:11:24.071494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.071458    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.078887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.078866    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.086099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.086078    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.093221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.093199    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.100248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.100229    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.107713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.107693    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.114831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.114810    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.115120 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.115101    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.115174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115130    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"} err="failed to get container status \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist"
Nov 01 06:11:24.115174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115147    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.115381 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.115364    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.115417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115388    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"} err="failed to get container status \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist"
Nov 01 06:11:24.115417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115404    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.115667 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.115649    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.115733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115672    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"} err="failed to get container status \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist"
Nov 01 06:11:24.115733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115684    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.115904 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.115889    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.115946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115907    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"} err="failed to get container status \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist"
Nov 01 06:11:24.115946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.115921    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.116144 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.116127    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.116206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116153    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"} err="failed to get container status \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist"
Nov 01 06:11:24.116206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116174    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.116397 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.116382    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.116431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116401    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"} err="failed to get container status \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist"
Nov 01 06:11:24.116431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116413    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.116633 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:24.116616    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.116700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116641    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"} err="failed to get container status \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist"
Nov 01 06:11:24.116700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116661    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.116941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116924    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"} err="failed to get container status \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist"
Nov 01 06:11:24.116989 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.116942    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.117152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117134    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"} err="failed to get container status \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist"
Nov 01 06:11:24.117213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117153    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.117395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117377    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"} err="failed to get container status \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist"
Nov 01 06:11:24.117442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117395    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.117675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117657    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"} err="failed to get container status \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist"
Nov 01 06:11:24.117720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117677    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.117904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117884    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"} err="failed to get container status \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist"
Nov 01 06:11:24.117946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.117905    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.118134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118105    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"} err="failed to get container status \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist"
Nov 01 06:11:24.118134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118133    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.118361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118342    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"} err="failed to get container status \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist"
Nov 01 06:11:24.118409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118362    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.118576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118559    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"} err="failed to get container status \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist"
Nov 01 06:11:24.118634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118576    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.118811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118795    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"} err="failed to get container status \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist"
Nov 01 06:11:24.118849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118812    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.119011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.118996    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"} err="failed to get container status \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist"
Nov 01 06:11:24.119048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119011    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.119208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119191    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"} err="failed to get container status \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist"
Nov 01 06:11:24.119249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119209    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.119410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119394    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"} err="failed to get container status \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist"
Nov 01 06:11:24.119447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119410    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.119656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119639    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"} err="failed to get container status \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist"
Nov 01 06:11:24.119701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119656    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.119830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119815    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"} err="failed to get container status \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist"
Nov 01 06:11:24.119830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119829    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.120011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.119992    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"} err="failed to get container status \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist"
Nov 01 06:11:24.120011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120005    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.120233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120215    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"} err="failed to get container status \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist"
Nov 01 06:11:24.120233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120233    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.120442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120422    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"} err="failed to get container status \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist"
Nov 01 06:11:24.120500 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120442    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.120684 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120665    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"} err="failed to get container status \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist"
Nov 01 06:11:24.120739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120685    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.120912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120897    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"} err="failed to get container status \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist"
Nov 01 06:11:24.120912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.120910    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.121117 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121100    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"} err="failed to get container status \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist"
Nov 01 06:11:24.121159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121117    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.121329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121315    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"} err="failed to get container status \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist"
Nov 01 06:11:24.121329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121329    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.121564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121546    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"} err="failed to get container status \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist"
Nov 01 06:11:24.121615 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121564    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.121766 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121751    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"} err="failed to get container status \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist"
Nov 01 06:11:24.121808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121767    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.122002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.121986    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"} err="failed to get container status \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist"
Nov 01 06:11:24.122002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122001    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.122206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122188    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"} err="failed to get container status \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist"
Nov 01 06:11:24.122206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122204    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.122424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122399    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"} err="failed to get container status \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist"
Nov 01 06:11:24.122424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122424    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.122666 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122650    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"} err="failed to get container status \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist"
Nov 01 06:11:24.122666 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122666    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.122891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122871    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"} err="failed to get container status \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist"
Nov 01 06:11:24.122891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.122891    4468 scope.go:122] "RemoveContainer" containerID="e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"
Nov 01 06:11:24.123083 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123068    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7"} err="failed to get container status \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": rpc error: code = NotFound desc = could not find container \"e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7\": container with ID starting with e0603630ae144ce517927cf3dcaa89cb0fd8037f8db55acbcd440817c12363a7 not found: ID does not exist"
Nov 01 06:11:24.123083 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123083    4468 scope.go:122] "RemoveContainer" containerID="07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"
Nov 01 06:11:24.123280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123264    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb"} err="failed to get container status \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": rpc error: code = NotFound desc = could not find container \"07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb\": container with ID starting with 07fbb8883057f3d23321eb9487e820cf4b81212c3b4e4ef71880869c4219b6bb not found: ID does not exist"
Nov 01 06:11:24.123325 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123279    4468 scope.go:122] "RemoveContainer" containerID="4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"
Nov 01 06:11:24.123517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123497    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e"} err="failed to get container status \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": rpc error: code = NotFound desc = could not find container \"4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e\": container with ID starting with 4ec116fe65b71a876b3538e427540e272c3b1af0c7ab7c0aa85d8864ab13243e not found: ID does not exist"
Nov 01 06:11:24.123561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123517    4468 scope.go:122] "RemoveContainer" containerID="3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"
Nov 01 06:11:24.123770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123751    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c"} err="failed to get container status \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": rpc error: code = NotFound desc = could not find container \"3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c\": container with ID starting with 3447761b8dc8a8fdc5ffb53f8a1dcc9a799d9426a09de6af037a0b6124d38c1c not found: ID does not exist"
Nov 01 06:11:24.123833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123771    4468 scope.go:122] "RemoveContainer" containerID="12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"
Nov 01 06:11:24.123997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123982    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4"} err="failed to get container status \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": rpc error: code = NotFound desc = could not find container \"12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4\": container with ID starting with 12fdde68f4290744b547b88993a77a8dfc785767af32f046c5792aaedbcc03c4 not found: ID does not exist"
Nov 01 06:11:24.123997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.123996    4468 scope.go:122] "RemoveContainer" containerID="3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"
Nov 01 06:11:24.124235 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.124216    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67"} err="failed to get container status \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": rpc error: code = NotFound desc = could not find container \"3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67\": container with ID starting with 3f938d327b600ebd92c1238e120eac4f0609f38de8f29688c09878b293abbf67 not found: ID does not exist"
Nov 01 06:11:24.124280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.124236    4468 scope.go:122] "RemoveContainer" containerID="7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"
Nov 01 06:11:24.124549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.124455    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5"} err="failed to get container status \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": rpc error: code = NotFound desc = could not find container \"7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5\": container with ID starting with 7af3950d3541b96266138b5d4cefc6f40000a53d0d5a80c7fee5925c5865e8e5 not found: ID does not exist"
Nov 01 06:11:24.225260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.225226    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:11:24.275352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.275316    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:11:24.276154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276137    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:11:24.276412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276400    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="init-config-reloader"
Nov 01 06:11:24.276443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276413    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="init-config-reloader"
Nov 01 06:11:24.276443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276426    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="prometheus"
Nov 01 06:11:24.276443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276431    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="prometheus"
Nov 01 06:11:24.276443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276437    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-web"
Nov 01 06:11:24.276443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276442    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-web"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276450    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-thanos"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276455    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-thanos"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276481    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="config-reloader"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276485    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="config-reloader"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276496    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276500    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276518    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="thanos-sidecar"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276523    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="thanos-sidecar"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276565    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="thanos-sidecar"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276573    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="prometheus"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276579    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-web"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276584    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy-thanos"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276590    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="config-reloader"
Nov 01 06:11:24.276653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.276597    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" containerName="kube-rbac-proxy"
Nov 01 06:11:24.280173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.280158    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.333651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333510    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 06:11:24.333651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333524    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 06:11:24.333651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333510    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 06:11:24.333651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333508    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333853    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333853    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-74g68\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333873    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333877    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.333882    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.334188    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 06:11:24.334211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.334213    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 06:11:24.334490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.334250    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b96qfv7d18d41\""
Nov 01 06:11:24.334490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.334284    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 06:11:24.334490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.334293    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 06:11:24.360392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360365    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-config\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360398    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrv2\" (UniqueName: \"kubernetes.io/projected/1cc29dcb-6cd2-44f2-9e45-327b1a320940-kube-api-access-bgrv2\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360424    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360532    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1cc29dcb-6cd2-44f2-9e45-327b1a320940-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360578    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360638    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360695 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360662    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360695 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360684    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360743    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1cc29dcb-6cd2-44f2-9e45-327b1a320940-config-out\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360761    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360819    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360846    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360864    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360914    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360945    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.360995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.360965    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.361028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.361011    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-web-config\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.361060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.361029    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.361092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.361071    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.361122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.361091    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.384253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.384224    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 06:11:24.384253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.384233    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 06:11:24.445947 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.445911    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:11:24.462386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.462356    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.462569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.462393    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.462569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.462417    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.462569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.462438    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1cc29dcb-6cd2-44f2-9e45-327b1a320940-config-out\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.462569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.462513    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.462569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.462545    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.463588    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.463661    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.463704    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.463862    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.463945    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.463999    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-web-config\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464142    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464177    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464214    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464256    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-config\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464519    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrv2\" (UniqueName: \"kubernetes.io/projected/1cc29dcb-6cd2-44f2-9e45-327b1a320940-kube-api-access-bgrv2\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464549    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464722    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1cc29dcb-6cd2-44f2-9e45-327b1a320940-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.464917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.464876    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.467114 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.467075    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.468250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.468024    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.468385    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1cc29dcb-6cd2-44f2-9e45-327b1a320940-config-out\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.469646    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.469887    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.470434    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.471106    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.471360    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.471951    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.472933    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.473649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.473252    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-web-config\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.474412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.474326    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.475424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.475377    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.476037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.475863    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.476037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.475999    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cc29dcb-6cd2-44f2-9e45-327b1a320940-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.476037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.476002    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.476306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.476285    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1cc29dcb-6cd2-44f2-9e45-327b1a320940-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.477392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.477371    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.477566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.477547    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1cc29dcb-6cd2-44f2-9e45-327b1a320940-config\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.528213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.528177    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrv2\" (UniqueName: \"kubernetes.io/projected/1cc29dcb-6cd2-44f2-9e45-327b1a320940-kube-api-access-bgrv2\") pod \"prometheus-k8s-0\" (UID: \"1cc29dcb-6cd2-44f2-9e45-327b1a320940\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:24.587409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.587325    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aad22ee-69b6-4fb0-8d9c-d6d4193288fc" path="/var/lib/kubelet/pods/0aad22ee-69b6-4fb0-8d9c-d6d4193288fc/volumes"
Nov 01 06:11:24.597551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:24.597525    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:25.023326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:25.023305    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 06:11:25.025561 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:11:25.025534    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc29dcb_6cd2_44f2_9e45_327b1a320940.slice/crio-e16b1d57667fb5e05d1442816245036238354dcd8757409dc6384bca0a003dac WatchSource:0}: Error finding container e16b1d57667fb5e05d1442816245036238354dcd8757409dc6384bca0a003dac: Status 404 returned error can't find the container with id e16b1d57667fb5e05d1442816245036238354dcd8757409dc6384bca0a003dac
Nov 01 06:11:25.065139 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:25.065109    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"e16b1d57667fb5e05d1442816245036238354dcd8757409dc6384bca0a003dac"}
Nov 01 06:11:26.070674 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:26.070637    4468 generic.go:356] "Generic (PLEG): container finished" podID="1cc29dcb-6cd2-44f2-9e45-327b1a320940" containerID="b183fa3c592077ceb26d2feb30ae88dc96c40045d8c8761a0b536c1a6cf89e53" exitCode=0
Nov 01 06:11:26.071061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:26.070720    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerDied","Data":"b183fa3c592077ceb26d2feb30ae88dc96c40045d8c8761a0b536c1a6cf89e53"}
Nov 01 06:11:27.077864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.077828    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"52df75ea0588639a2f97733ba115701bfd310281db8c0b9dcb25c54181e77706"}
Nov 01 06:11:27.078351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.077890    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"ded3fc0721a5b8c5db6945c53e4bf9005ea5d7e2e93d4a5cc053dad737970c84"}
Nov 01 06:11:27.078351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.077917    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"e0ea6db2987cd237f688376c0f010b8054a369654aa1e921694a1fb9972642b7"}
Nov 01 06:11:27.078351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.077940    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"eb06e5efc76fc2c0269d9e0ce7afad327b887ba46ac523849de474bac0c48d80"}
Nov 01 06:11:27.078351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.077963    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"834c16ae8b0ac8d8139c8c30822c8b33d69243aa6ffb295420c6b77aafeed5ed"}
Nov 01 06:11:27.078351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.077985    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1cc29dcb-6cd2-44f2-9e45-327b1a320940","Type":"ContainerStarted","Data":"fc4876babe70ce2d138c851d69b88998735a09008c970293df4c35c425863f8f"}
Nov 01 06:11:27.201937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:27.201885    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.185679823 podStartE2EDuration="3.201871021s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.191198ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:27.200617066 +0000 UTC m=+98.768423921" watchObservedRunningTime="2026-11-01 06:11:27.201871021 +0000 UTC m=+98.769677850"
Nov 01 06:11:29.598732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:29.598688    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:11:31.201177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.201132    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fd5bc7774-m4tch"]
Nov 01 06:11:31.203645 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.203622    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232163    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25364f5-aecd-40df-830f-0de982a6d231-tmp\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232230    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e25364f5-aecd-40df-830f-0de982a6d231-console-serving-cert\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232280    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-trusted-ca-bundle\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232309    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-oauth-serving-cert\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232331    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-console-config\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232654 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232481    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e25364f5-aecd-40df-830f-0de982a6d231-console-oauth-config\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232654 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232507    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-service-ca\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.232654 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.232533    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcrt\" (UniqueName: \"kubernetes.io/projected/e25364f5-aecd-40df-830f-0de982a6d231-kube-api-access-kbcrt\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.306068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.306028    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd5bc7774-m4tch"]
Nov 01 06:11:31.333408 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333374    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e25364f5-aecd-40df-830f-0de982a6d231-console-serving-cert\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333408 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333414    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-trusted-ca-bundle\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333447    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-oauth-serving-cert\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333520    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-console-config\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333600    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e25364f5-aecd-40df-830f-0de982a6d231-console-oauth-config\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333619    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-service-ca\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333639    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcrt\" (UniqueName: \"kubernetes.io/projected/e25364f5-aecd-40df-830f-0de982a6d231-kube-api-access-kbcrt\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.333872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.333674    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25364f5-aecd-40df-830f-0de982a6d231-tmp\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.334308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.334286    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25364f5-aecd-40df-830f-0de982a6d231-tmp\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.334631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.334600    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-service-ca\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.334697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.334600    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-trusted-ca-bundle\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.334697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.334667    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-oauth-serving-cert\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.334866 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.334845    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e25364f5-aecd-40df-830f-0de982a6d231-console-config\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.336383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.336352    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e25364f5-aecd-40df-830f-0de982a6d231-console-oauth-config\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.336504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.336358    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e25364f5-aecd-40df-830f-0de982a6d231-console-serving-cert\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.389813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.389779    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcrt\" (UniqueName: \"kubernetes.io/projected/e25364f5-aecd-40df-830f-0de982a6d231-kube-api-access-kbcrt\") pod \"console-5fd5bc7774-m4tch\" (UID: \"e25364f5-aecd-40df-830f-0de982a6d231\") " pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.515100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.515014    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:31.882668 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:31.882633    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd5bc7774-m4tch"]
Nov 01 06:11:31.885758 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:11:31.885728    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25364f5_aecd_40df_830f_0de982a6d231.slice/crio-76a7e7e9cd62abeb3a0f963a79cedc150989ac2984a43fceec3ecdaed25db2af WatchSource:0}: Error finding container 76a7e7e9cd62abeb3a0f963a79cedc150989ac2984a43fceec3ecdaed25db2af: Status 404 returned error can't find the container with id 76a7e7e9cd62abeb3a0f963a79cedc150989ac2984a43fceec3ecdaed25db2af
Nov 01 06:11:32.099176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:32.099143    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd5bc7774-m4tch" event={"ID":"e25364f5-aecd-40df-830f-0de982a6d231","Type":"ContainerStarted","Data":"e58bf661d8bab92826dbd08b97d5e06f1cd1582862e5142fc145ffbd36d4f1c0"}
Nov 01 06:11:32.099360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:32.099194    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd5bc7774-m4tch" event={"ID":"e25364f5-aecd-40df-830f-0de982a6d231","Type":"ContainerStarted","Data":"76a7e7e9cd62abeb3a0f963a79cedc150989ac2984a43fceec3ecdaed25db2af"}
Nov 01 06:11:32.214801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:32.214701    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fd5bc7774-m4tch" podStartSLOduration=1.214687596 podStartE2EDuration="1.214687596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:32.213426654 +0000 UTC m=+103.781233502" watchObservedRunningTime="2026-11-01 06:11:32.214687596 +0000 UTC m=+103.782494423"
Nov 01 06:11:32.947746 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:11:32.947709    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:11:41.515728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:41.515686    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:41.516213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:41.515789    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:41.520606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:41.520583    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:42.142354 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:42.142322    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fd5bc7774-m4tch"
Nov 01 06:11:42.306535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:11:42.306493    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f8fb769-njhkb"]
Nov 01 06:12:02.946817 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:12:02.946738    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:12:07.326897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.326831    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86f8fb769-njhkb" podUID="029a78e5-774b-4d43-a8ac-cfbfa67f5c93" containerName="console" containerID="cri-o://24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093" gracePeriod=15
Nov 01 06:12:07.584352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.584294    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:12:07.640903 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.640870    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.640928    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp\" (UniqueName: \"kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.640951    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x\" (UniqueName: \"kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.640968    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.640984    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641010    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641268 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641083    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641268 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641109    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert\") pod \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\" (UID: \"029a78e5-774b-4d43-a8ac-cfbfa67f5c93\") "
Nov 01 06:12:07.641370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641255    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:07.641432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641358    4468 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-tmp\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.641752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641657    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:07.641752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641711    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:07.641752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641722    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:07.641995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.641961    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:07.643333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.643308    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:12:07.643433 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.643396    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "kube-api-access-xnl4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:07.643598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.643577    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config" pod "029a78e5-774b-4d43-a8ac-cfbfa67f5c93" (UID: "029a78e5-774b-4d43-a8ac-cfbfa67f5c93"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:12:07.742078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742040    4468 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-trusted-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.742078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742075    4468 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-oauth-serving-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.742289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742089    4468 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-serving-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.742289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742101    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnl4x\" (UniqueName: \"kubernetes.io/projected/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-kube-api-access-xnl4x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.742289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742116    4468 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.742289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742128    4468 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-oauth-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:07.742289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:07.742139    4468 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/029a78e5-774b-4d43-a8ac-cfbfa67f5c93-console-config\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:08.242159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.242118    4468 generic.go:356] "Generic (PLEG): container finished" podID="029a78e5-774b-4d43-a8ac-cfbfa67f5c93" containerID="24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093" exitCode=0
Nov 01 06:12:08.242350 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.242167    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f8fb769-njhkb"
Nov 01 06:12:08.242350 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.242174    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f8fb769-njhkb" event={"ID":"029a78e5-774b-4d43-a8ac-cfbfa67f5c93","Type":"ContainerDied","Data":"24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093"}
Nov 01 06:12:08.242350 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.242211    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f8fb769-njhkb" event={"ID":"029a78e5-774b-4d43-a8ac-cfbfa67f5c93","Type":"ContainerDied","Data":"7f1b79b1a0f8c768f36f56fa50c355de45868da167148d2287d6348cace6d7f8"}
Nov 01 06:12:08.242350 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.242230    4468 scope.go:122] "RemoveContainer" containerID="24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093"
Nov 01 06:12:08.251399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.251381    4468 scope.go:122] "RemoveContainer" containerID="24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093"
Nov 01 06:12:08.251683 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:12:08.251661    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093\": container with ID starting with 24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093 not found: ID does not exist" containerID="24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093"
Nov 01 06:12:08.251734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.251692    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093"} err="failed to get container status \"24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093\": rpc error: code = NotFound desc = could not find container \"24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093\": container with ID starting with 24e7773bc8a2fcc0e5a2d8d882d40ca91fa344e467511116b3e344c42c91f093 not found: ID does not exist"
Nov 01 06:12:08.404089 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.404055    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f8fb769-njhkb"]
Nov 01 06:12:08.453836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.453801    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86f8fb769-njhkb"]
Nov 01 06:12:08.591674 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:08.590191    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029a78e5-774b-4d43-a8ac-cfbfa67f5c93" path="/var/lib/kubelet/pods/029a78e5-774b-4d43-a8ac-cfbfa67f5c93/volumes"
Nov 01 06:12:24.598606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:24.598558    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:12:24.615076 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:24.615043    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:12:25.320226 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:12:25.320198    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 06:12:32.946693 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:12:32.946655    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:13:02.947109 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:13:02.947072    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:13:32.946617 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:13:32.946576    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:14:02.946954 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:14:02.946917    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:14:21.725233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.725195    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"]
Nov 01 06:14:21.727649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.725569    4468 cpu_manager.go:414] "containerMap: removing container" podUID="029a78e5-774b-4d43-a8ac-cfbfa67f5c93" containerName="console"
Nov 01 06:14:21.727649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.725586    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029a78e5-774b-4d43-a8ac-cfbfa67f5c93" containerName="console"
Nov 01 06:14:21.727649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.725664    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="029a78e5-774b-4d43-a8ac-cfbfa67f5c93" containerName="console"
Nov 01 06:14:21.728694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.728675    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.782730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.782691    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7nc\" (UniqueName: \"kubernetes.io/projected/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-kube-api-access-dw7nc\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.782730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.782734    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.782934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.782755    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.782934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.782886    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.782934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.782918    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.783030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.782981    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.860911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.860870    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 06:14:21.884409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884366    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7nc\" (UniqueName: \"kubernetes.io/projected/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-kube-api-access-dw7nc\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.884584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884415    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.884584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884439    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.884584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884505    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.884698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884590    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.884698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884645    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.884884 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.884864    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.886850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.886825    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.905799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.905766    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 06:14:21.905799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.905791    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-qjjpq\""
Nov 01 06:14:21.905990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.905780    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 06:14:21.905990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.905770    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 06:14:21.905990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.905825    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 06:14:21.917540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.917515    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.917694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.917565    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.917694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.917653    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.941299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.941265    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7nc\" (UniqueName: \"kubernetes.io/projected/b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9-kube-api-access-dw7nc\") pod \"opendatahub-operator-controller-manager-5965b9bff7-hjg5w\" (UID: \"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:21.956170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:21.956136    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"]
Nov 01 06:14:22.043443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:22.043356    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:22.528353 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:22.528320    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ceb5db_e5c5_4159_a55f_1689bb4fb9e9.slice/crio-a2210c7d28dc8cfa672c3bc8beab21d03f57db882a225ec1a88b1a0a45cddcb3 WatchSource:0}: Error finding container a2210c7d28dc8cfa672c3bc8beab21d03f57db882a225ec1a88b1a0a45cddcb3: Status 404 returned error can't find the container with id a2210c7d28dc8cfa672c3bc8beab21d03f57db882a225ec1a88b1a0a45cddcb3
Nov 01 06:14:22.528543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:22.528376    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"]
Nov 01 06:14:22.736325 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:22.736293    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w" event={"ID":"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9","Type":"ContainerStarted","Data":"a2210c7d28dc8cfa672c3bc8beab21d03f57db882a225ec1a88b1a0a45cddcb3"}
Nov 01 06:14:26.755274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:26.755235    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w" event={"ID":"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9","Type":"ContainerStarted","Data":"4941905fe13cc1ab5349bf7e77e01e521b007e13ca4ea54db5b3703b6561c58d"}
Nov 01 06:14:27.761774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:27.761733    4468 generic.go:356] "Generic (PLEG): container finished" podID="b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9" containerID="4941905fe13cc1ab5349bf7e77e01e521b007e13ca4ea54db5b3703b6561c58d" exitCode=0
Nov 01 06:14:27.762173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:27.761818    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w" event={"ID":"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9","Type":"ContainerDied","Data":"4941905fe13cc1ab5349bf7e77e01e521b007e13ca4ea54db5b3703b6561c58d"}
Nov 01 06:14:28.767971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:28.767935    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w" event={"ID":"b9ceb5db-e5c5-4159-a55f-1689bb4fb9e9","Type":"ContainerStarted","Data":"9a69affb1ad71a5210d22656fc927d2711ec6a03441f257db65b2e49d7cf5a40"}
Nov 01 06:14:28.768390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:28.768063    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:29.030957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:29.030900    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w" podStartSLOduration=2.659805727 podStartE2EDuration="8.030883469s" totalImagesPullingTime="4.884677985s" totalInitContainerRuntime="486.399757ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:29.02959801 +0000 UTC m=+280.597404840" watchObservedRunningTime="2026-11-01 06:14:29.030883469 +0000 UTC m=+280.598690300"
Nov 01 06:14:31.920111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:31.920082    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"]
Nov 01 06:14:31.923204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:31.923187    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:31.980217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:31.980183    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-57xlm\""
Nov 01 06:14:31.981145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:31.981115    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8m74\" (UniqueName: \"kubernetes.io/projected/eced03d9-a296-478b-b46e-086f66419f65-kube-api-access-b8m74\") pod \"jobset-operator-674cd59dc5-thpdh\" (UID: \"eced03d9-a296-478b-b46e-086f66419f65\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:31.981777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:31.981228    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eced03d9-a296-478b-b46e-086f66419f65-tmp\") pod \"jobset-operator-674cd59dc5-thpdh\" (UID: \"eced03d9-a296-478b-b46e-086f66419f65\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:32.029412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.029370    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 06:14:32.029412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.029386    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 06:14:32.030710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.030686    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"]
Nov 01 06:14:32.082410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.082378    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8m74\" (UniqueName: \"kubernetes.io/projected/eced03d9-a296-478b-b46e-086f66419f65-kube-api-access-b8m74\") pod \"jobset-operator-674cd59dc5-thpdh\" (UID: \"eced03d9-a296-478b-b46e-086f66419f65\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:32.082582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.082513    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eced03d9-a296-478b-b46e-086f66419f65-tmp\") pod \"jobset-operator-674cd59dc5-thpdh\" (UID: \"eced03d9-a296-478b-b46e-086f66419f65\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:32.082902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.082882    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eced03d9-a296-478b-b46e-086f66419f65-tmp\") pod \"jobset-operator-674cd59dc5-thpdh\" (UID: \"eced03d9-a296-478b-b46e-086f66419f65\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:32.138854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.138816    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8m74\" (UniqueName: \"kubernetes.io/projected/eced03d9-a296-478b-b46e-086f66419f65-kube-api-access-b8m74\") pod \"jobset-operator-674cd59dc5-thpdh\" (UID: \"eced03d9-a296-478b-b46e-086f66419f65\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:32.248237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.248144    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"
Nov 01 06:14:32.621522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.621499    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh"]
Nov 01 06:14:32.624503 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:32.624459    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeced03d9_a296_478b_b46e_086f66419f65.slice/crio-18aab5b9cbc6b4a617c072a104bda1de6532d527209dd8caffd498e6187d2fa7 WatchSource:0}: Error finding container 18aab5b9cbc6b4a617c072a104bda1de6532d527209dd8caffd498e6187d2fa7: Status 404 returned error can't find the container with id 18aab5b9cbc6b4a617c072a104bda1de6532d527209dd8caffd498e6187d2fa7
Nov 01 06:14:32.785962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:32.785926    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh" event={"ID":"eced03d9-a296-478b-b46e-086f66419f65","Type":"ContainerStarted","Data":"18aab5b9cbc6b4a617c072a104bda1de6532d527209dd8caffd498e6187d2fa7"}
Nov 01 06:14:32.946263 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:14:32.946176    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:14:36.804391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:36.804347    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh" event={"ID":"eced03d9-a296-478b-b46e-086f66419f65","Type":"ContainerStarted","Data":"f76e1f7be22405f650f25a0b6a625ab151a745beee49c39bf335c1ae6784bc76"}
Nov 01 06:14:39.774621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:39.774592    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-hjg5w"
Nov 01 06:14:40.065685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:40.065625    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-thpdh" podStartSLOduration=5.410393941 podStartE2EDuration="9.065604088s" totalImagesPullingTime="3.655210147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:36.938258627 +0000 UTC m=+288.506065459" watchObservedRunningTime="2026-11-01 06:14:40.065604088 +0000 UTC m=+291.633410919"
Nov 01 06:14:46.823910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.823874    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"]
Nov 01 06:14:46.828834 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.828811    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:46.889062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.889031    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 06:14:46.915838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.915796    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fxz\" (UniqueName: \"kubernetes.io/projected/56dbba94-0631-4315-949b-563a19b67a26-kube-api-access-87fxz\") pod \"cert-manager-cainjector-5c57bbd4fc-4g7gv\" (UID: \"56dbba94-0631-4315-949b-563a19b67a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:46.916033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.915942    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56dbba94-0631-4315-949b-563a19b67a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4g7gv\" (UID: \"56dbba94-0631-4315-949b-563a19b67a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:46.938798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.938764    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 06:14:46.938925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.938808    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dsl28\""
Nov 01 06:14:46.940240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:46.940215    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"]
Nov 01 06:14:47.016943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.016903    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56dbba94-0631-4315-949b-563a19b67a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4g7gv\" (UID: \"56dbba94-0631-4315-949b-563a19b67a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:47.016943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.016947    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fxz\" (UniqueName: \"kubernetes.io/projected/56dbba94-0631-4315-949b-563a19b67a26-kube-api-access-87fxz\") pod \"cert-manager-cainjector-5c57bbd4fc-4g7gv\" (UID: \"56dbba94-0631-4315-949b-563a19b67a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:47.075964 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.075891    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56dbba94-0631-4315-949b-563a19b67a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4g7gv\" (UID: \"56dbba94-0631-4315-949b-563a19b67a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:47.076091 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.076032    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fxz\" (UniqueName: \"kubernetes.io/projected/56dbba94-0631-4315-949b-563a19b67a26-kube-api-access-87fxz\") pod \"cert-manager-cainjector-5c57bbd4fc-4g7gv\" (UID: \"56dbba94-0631-4315-949b-563a19b67a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:47.139042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.139005    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"
Nov 01 06:14:47.516948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.516742    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv"]
Nov 01 06:14:47.519522 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:47.519492    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56dbba94_0631_4315_949b_563a19b67a26.slice/crio-2095dd50ec328a85f6392610dd5e070656228d7a66f5f16218ea9d9eb3d00e44 WatchSource:0}: Error finding container 2095dd50ec328a85f6392610dd5e070656228d7a66f5f16218ea9d9eb3d00e44: Status 404 returned error can't find the container with id 2095dd50ec328a85f6392610dd5e070656228d7a66f5f16218ea9d9eb3d00e44
Nov 01 06:14:47.852770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:47.852732    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv" event={"ID":"56dbba94-0631-4315-949b-563a19b67a26","Type":"ContainerStarted","Data":"2095dd50ec328a85f6392610dd5e070656228d7a66f5f16218ea9d9eb3d00e44"}
Nov 01 06:14:51.871959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:51.871916    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv" event={"ID":"56dbba94-0631-4315-949b-563a19b67a26","Type":"ContainerStarted","Data":"b7f23d97d197764593209e3a9e18f835bf777a4ec6fbd62bee1577addea89719"}
Nov 01 06:14:51.989419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:51.989357    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4g7gv" podStartSLOduration=1.75501434 podStartE2EDuration="5.989340079s" totalImagesPullingTime="4.234325739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:51.988377231 +0000 UTC m=+303.556184072" watchObservedRunningTime="2026-11-01 06:14:51.989340079 +0000 UTC m=+303.557146908"
Nov 01 06:14:52.088216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.088177    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"]
Nov 01 06:14:52.091445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.091422    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.143837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.143804    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6zsdn\""
Nov 01 06:14:52.148975 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.148946    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"]
Nov 01 06:14:52.163702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.163667    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgcc4\" (UniqueName: \"kubernetes.io/projected/e9aeb556-5c1f-4220-ab50-36824ef4754a-kube-api-access-hgcc4\") pod \"cert-manager-webhook-5dbbbf75cc-rxfxj\" (UID: \"e9aeb556-5c1f-4220-ab50-36824ef4754a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.163882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.163710    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9aeb556-5c1f-4220-ab50-36824ef4754a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-rxfxj\" (UID: \"e9aeb556-5c1f-4220-ab50-36824ef4754a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.264309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.264268    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcc4\" (UniqueName: \"kubernetes.io/projected/e9aeb556-5c1f-4220-ab50-36824ef4754a-kube-api-access-hgcc4\") pod \"cert-manager-webhook-5dbbbf75cc-rxfxj\" (UID: \"e9aeb556-5c1f-4220-ab50-36824ef4754a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.264309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.264315    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9aeb556-5c1f-4220-ab50-36824ef4754a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-rxfxj\" (UID: \"e9aeb556-5c1f-4220-ab50-36824ef4754a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.322909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.322875    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9aeb556-5c1f-4220-ab50-36824ef4754a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-rxfxj\" (UID: \"e9aeb556-5c1f-4220-ab50-36824ef4754a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.323070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.322944    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcc4\" (UniqueName: \"kubernetes.io/projected/e9aeb556-5c1f-4220-ab50-36824ef4754a-kube-api-access-hgcc4\") pod \"cert-manager-webhook-5dbbbf75cc-rxfxj\" (UID: \"e9aeb556-5c1f-4220-ab50-36824ef4754a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.402713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.402627    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:52.775416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.775336    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"]
Nov 01 06:14:52.778815 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:52.778783    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9aeb556_5c1f_4220_ab50_36824ef4754a.slice/crio-c253a0429895ebca7489c7dff40fa54282462261655c13411238336dfb7bbd15 WatchSource:0}: Error finding container c253a0429895ebca7489c7dff40fa54282462261655c13411238336dfb7bbd15: Status 404 returned error can't find the container with id c253a0429895ebca7489c7dff40fa54282462261655c13411238336dfb7bbd15
Nov 01 06:14:52.883753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.883707    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj" event={"ID":"e9aeb556-5c1f-4220-ab50-36824ef4754a","Type":"ContainerStarted","Data":"c431054426a940f3692d2c8eb17f2a99dcfe0e7099866095fedef5ededd97893"}
Nov 01 06:14:52.884234 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:52.883763    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj" event={"ID":"e9aeb556-5c1f-4220-ab50-36824ef4754a","Type":"ContainerStarted","Data":"c253a0429895ebca7489c7dff40fa54282462261655c13411238336dfb7bbd15"}
Nov 01 06:14:53.001715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.001662    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj" podStartSLOduration=1.001648005 podStartE2EDuration="1.001648005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:53.000525888 +0000 UTC m=+304.568332749" watchObservedRunningTime="2026-11-01 06:14:53.001648005 +0000 UTC m=+304.569454835"
Nov 01 06:14:53.461363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.461327    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"]
Nov 01 06:14:53.464934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.464912    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.517397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.517363    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 06:14:53.520190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.520171    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 06:14:53.568403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.568365    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"]
Nov 01 06:14:53.580210 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.580177    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2673628-ba47-467c-a366-ee01f7c3036d-tls-certs\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.580389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.580223    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27txp\" (UniqueName: \"kubernetes.io/projected/c2673628-ba47-467c-a366-ee01f7c3036d-kube-api-access-27txp\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.580389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.580313    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2673628-ba47-467c-a366-ee01f7c3036d-tmp\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.584018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.583993    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 06:14:53.584132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.584072    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hhrjq\""
Nov 01 06:14:53.584223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.584208    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 06:14:53.681053 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.681017    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2673628-ba47-467c-a366-ee01f7c3036d-tls-certs\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.681229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.681061    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27txp\" (UniqueName: \"kubernetes.io/projected/c2673628-ba47-467c-a366-ee01f7c3036d-kube-api-access-27txp\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.681229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.681089    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2673628-ba47-467c-a366-ee01f7c3036d-tmp\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.683606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.683570    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2673628-ba47-467c-a366-ee01f7c3036d-tmp\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.683772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.683751    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2673628-ba47-467c-a366-ee01f7c3036d-tls-certs\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.742213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.742131    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27txp\" (UniqueName: \"kubernetes.io/projected/c2673628-ba47-467c-a366-ee01f7c3036d-kube-api-access-27txp\") pod \"kube-auth-proxy-68cbf89484-cmqtk\" (UID: \"c2673628-ba47-467c-a366-ee01f7c3036d\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.777691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.777657    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"
Nov 01 06:14:53.887698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:53.887665    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:14:54.168437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:54.168408    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk"]
Nov 01 06:14:54.170031 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:54.169994    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2673628_ba47_467c_a366_ee01f7c3036d.slice/crio-a646e455a41a6d425eca3216b2195eb048257930a5217e52d2a37b2347236470 WatchSource:0}: Error finding container a646e455a41a6d425eca3216b2195eb048257930a5217e52d2a37b2347236470: Status 404 returned error can't find the container with id a646e455a41a6d425eca3216b2195eb048257930a5217e52d2a37b2347236470
Nov 01 06:14:54.172095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:54.172077    4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:14:54.893617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:54.893583    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk" event={"ID":"c2673628-ba47-467c-a366-ee01f7c3036d","Type":"ContainerStarted","Data":"a646e455a41a6d425eca3216b2195eb048257930a5217e52d2a37b2347236470"}
Nov 01 06:14:55.913124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.913085    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:14:55.916770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.916748    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:55.972639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.971829    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 06:14:55.972639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.971946    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s9fd7\""
Nov 01 06:14:55.972639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.971999    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 06:14:55.972639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.972232    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 06:14:55.973858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:55.973836    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:14:56.004297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.004267    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file\") pod \"limitador-limitador-74dfcfb894-62f4s\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.004491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.004304    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4fh\" (UniqueName: \"kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh\") pod \"limitador-limitador-74dfcfb894-62f4s\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.105454 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.105422    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file\") pod \"limitador-limitador-74dfcfb894-62f4s\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.105642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.105477    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4fh\" (UniqueName: \"kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh\") pod \"limitador-limitador-74dfcfb894-62f4s\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.106564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.106532    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file\") pod \"limitador-limitador-74dfcfb894-62f4s\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.165956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.165864    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:14:56.168335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.168313    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4fh\" (UniqueName: \"kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh\") pod \"limitador-limitador-74dfcfb894-62f4s\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.233373 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.233337    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:14:56.628187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.628152    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:14:56.632615 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:56.632583    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eae2df_3ede_4cf2_a2e8_f8df8f1e45dc.slice/crio-7795b9051fce951a628dcf3da2cbd82d9913472b3add296ed8b6ffb84c5d8cfd WatchSource:0}: Error finding container 7795b9051fce951a628dcf3da2cbd82d9913472b3add296ed8b6ffb84c5d8cfd: Status 404 returned error can't find the container with id 7795b9051fce951a628dcf3da2cbd82d9913472b3add296ed8b6ffb84c5d8cfd
Nov 01 06:14:56.904561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:56.904438    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s" event={"ID":"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc","Type":"ContainerStarted","Data":"7795b9051fce951a628dcf3da2cbd82d9913472b3add296ed8b6ffb84c5d8cfd"}
Nov 01 06:14:57.396133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.396103    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:14:57.399511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.399489    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.456925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.456842    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:14:57.521079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.521047    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4l9p\" (UniqueName: \"kubernetes.io/projected/81cbf762-f2c8-44ae-b2a5-e1a8e7166be8-kube-api-access-j4l9p\") pod \"limitador-limitador-7c9b87d574-b57dc\" (UID: \"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.521279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.521186    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81cbf762-f2c8-44ae-b2a5-e1a8e7166be8-config-file\") pod \"limitador-limitador-7c9b87d574-b57dc\" (UID: \"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.622020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.621978    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81cbf762-f2c8-44ae-b2a5-e1a8e7166be8-config-file\") pod \"limitador-limitador-7c9b87d574-b57dc\" (UID: \"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.622197 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.622028    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4l9p\" (UniqueName: \"kubernetes.io/projected/81cbf762-f2c8-44ae-b2a5-e1a8e7166be8-kube-api-access-j4l9p\") pod \"limitador-limitador-7c9b87d574-b57dc\" (UID: \"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.622615 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.622596    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81cbf762-f2c8-44ae-b2a5-e1a8e7166be8-config-file\") pod \"limitador-limitador-7c9b87d574-b57dc\" (UID: \"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.678459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.678416    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4l9p\" (UniqueName: \"kubernetes.io/projected/81cbf762-f2c8-44ae-b2a5-e1a8e7166be8-kube-api-access-j4l9p\") pod \"limitador-limitador-7c9b87d574-b57dc\" (UID: \"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.694413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.694368    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:14:57.711837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.711755    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:14:57.910511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:57.910434    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk" event={"ID":"c2673628-ba47-467c-a366-ee01f7c3036d","Type":"ContainerStarted","Data":"64ea74c45c232d6244a35b098106318f1fa98f81c5548eccd0410e99e4c39bef"}
Nov 01 06:14:58.026879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:58.026756    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68cbf89484-cmqtk" podStartSLOduration=2.088694274 podStartE2EDuration="5.026740061s" totalImagesPullingTime="2.938045787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:58.025308211 +0000 UTC m=+309.593115063" watchObservedRunningTime="2026-11-01 06:14:58.026740061 +0000 UTC m=+309.594546891"
Nov 01 06:14:58.088912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:58.088849    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:14:58.091854 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:14:58.091816    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cbf762_f2c8_44ae_b2a5_e1a8e7166be8.slice/crio-41471f1d58783d4cea409644335ce1eef714b927db141ac66cf2eeca165e9516 WatchSource:0}: Error finding container 41471f1d58783d4cea409644335ce1eef714b927db141ac66cf2eeca165e9516: Status 404 returned error can't find the container with id 41471f1d58783d4cea409644335ce1eef714b927db141ac66cf2eeca165e9516
Nov 01 06:14:58.916762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:58.916729    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc" event={"ID":"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8","Type":"ContainerStarted","Data":"41471f1d58783d4cea409644335ce1eef714b927db141ac66cf2eeca165e9516"}
Nov 01 06:14:59.896335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:14:59.896306    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rxfxj"
Nov 01 06:15:00.926974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:00.926937    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s" event={"ID":"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc","Type":"ContainerStarted","Data":"a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8"}
Nov 01 06:15:00.927500 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:00.927038    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:15:00.928576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:00.928555    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc" event={"ID":"81cbf762-f2c8-44ae-b2a5-e1a8e7166be8","Type":"ContainerStarted","Data":"edb8c8c875851adedd47e840530071eea1c544c9ee9f97854b82e41382cbeff7"}
Nov 01 06:15:00.928698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:00.928685    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:15:01.047990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:01.047932    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s" podStartSLOduration=2.643399154 podStartE2EDuration="6.047916673s" totalImagesPullingTime="3.404517519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.046359526 +0000 UTC m=+312.614166357" watchObservedRunningTime="2026-11-01 06:15:01.047916673 +0000 UTC m=+312.615723546"
Nov 01 06:15:01.162688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:01.162632    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc" podStartSLOduration=2.07225633 podStartE2EDuration="4.162613841s" totalImagesPullingTime="2.090357511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.161455108 +0000 UTC m=+312.729261951" watchObservedRunningTime="2026-11-01 06:15:01.162613841 +0000 UTC m=+312.730420670"
Nov 01 06:15:02.948206 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:15:02.948173    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:15:11.933772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:11.933743    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-b57dc"
Nov 01 06:15:11.934191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:11.933813    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:15:12.097366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.097328    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:15:12.097584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.097519    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s" podUID="f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" containerName="limitador" containerID="cri-o://a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8" gracePeriod=30
Nov 01 06:15:12.640710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.640681    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:15:12.659898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.659815    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file\" (UniqueName: \"kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file\") pod \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") "
Nov 01 06:15:12.659898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.659877    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh\" (UniqueName: \"kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh\") pod \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\" (UID: \"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc\") "
Nov 01 06:15:12.660232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.660211    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file" pod "f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" (UID: "f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:12.662393 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.662365    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh" pod "f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" (UID: "f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc"). InnerVolumeSpecName "kube-api-access-sb4fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:12.760755 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.760725    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb4fh\" (UniqueName: \"kubernetes.io/projected/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-kube-api-access-sb4fh\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:12.760755 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.760754    4468 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc-config-file\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:12.982690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.982607    4468 generic.go:356] "Generic (PLEG): container finished" podID="f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" containerID="a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8" exitCode=0
Nov 01 06:15:12.982690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.982660    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s"
Nov 01 06:15:12.982690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.982682    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s" event={"ID":"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc","Type":"ContainerDied","Data":"a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8"}
Nov 01 06:15:12.983191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.982734    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-62f4s" event={"ID":"f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc","Type":"ContainerDied","Data":"7795b9051fce951a628dcf3da2cbd82d9913472b3add296ed8b6ffb84c5d8cfd"}
Nov 01 06:15:12.983191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.982763    4468 scope.go:122] "RemoveContainer" containerID="a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8"
Nov 01 06:15:12.991509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.991487    4468 scope.go:122] "RemoveContainer" containerID="a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8"
Nov 01 06:15:12.991834 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:15:12.991814    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8\": container with ID starting with a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8 not found: ID does not exist" containerID="a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8"
Nov 01 06:15:12.991907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:12.991843    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8"} err="failed to get container status \"a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8\": rpc error: code = NotFound desc = could not find container \"a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8\": container with ID starting with a8ae359ca5f1ee7df60de1fc58238e6e7a0b13afdc9376b326bbac29d24c69a8 not found: ID does not exist"
Nov 01 06:15:13.172205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:13.172167    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:15:13.221185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:13.221149    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-62f4s"]
Nov 01 06:15:14.588067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:14.588026    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" path="/var/lib/kubelet/pods/f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc/volumes"
Nov 01 06:15:20.118713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.118677    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"]
Nov 01 06:15:20.119116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.119028    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" containerName="limitador"
Nov 01 06:15:20.119116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.119039    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" containerName="limitador"
Nov 01 06:15:20.119116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.119115    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9eae2df-3ede-4cf2-a2e8-f8df8f1e45dc" containerName="limitador"
Nov 01 06:15:20.123274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.123254    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.175439 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.175399    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-256pz\""
Nov 01 06:15:20.175439 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.175447    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 06:15:20.225154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.225117    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 06:15:20.229859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.229827    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/409da98c-561b-4be9-99cc-6535a33b667f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.229859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.229863    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/409da98c-561b-4be9-99cc-6535a33b667f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.230111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.229935    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wfq\" (UniqueName: \"kubernetes.io/projected/409da98c-561b-4be9-99cc-6535a33b667f-kube-api-access-g2wfq\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.230111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.229964    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/409da98c-561b-4be9-99cc-6535a33b667f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.325545 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.325503    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 06:15:20.329561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.329527    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"]
Nov 01 06:15:20.336158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.335451    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/409da98c-561b-4be9-99cc-6535a33b667f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.336158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.335517    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/409da98c-561b-4be9-99cc-6535a33b667f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.336158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.335575    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wfq\" (UniqueName: \"kubernetes.io/projected/409da98c-561b-4be9-99cc-6535a33b667f-kube-api-access-g2wfq\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.336158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.335620    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/409da98c-561b-4be9-99cc-6535a33b667f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.336158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.336061    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/409da98c-561b-4be9-99cc-6535a33b667f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.336488 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.336277    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/409da98c-561b-4be9-99cc-6535a33b667f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.337198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.337170    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/409da98c-561b-4be9-99cc-6535a33b667f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.393979 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.393904    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wfq\" (UniqueName: \"kubernetes.io/projected/409da98c-561b-4be9-99cc-6535a33b667f-kube-api-access-g2wfq\") pod \"kserve-module-controller-manager-7566cf5fc8-pwf6c\" (UID: \"409da98c-561b-4be9-99cc-6535a33b667f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.435761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.435717    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:20.858012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:20.857982    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"]
Nov 01 06:15:20.860404 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:15:20.860373    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409da98c_561b_4be9_99cc_6535a33b667f.slice/crio-ce32c8759c9783d01c021be4a29a8f296ef69e14d047b28e9de3a32b44a5300c WatchSource:0}: Error finding container ce32c8759c9783d01c021be4a29a8f296ef69e14d047b28e9de3a32b44a5300c: Status 404 returned error can't find the container with id ce32c8759c9783d01c021be4a29a8f296ef69e14d047b28e9de3a32b44a5300c
Nov 01 06:15:21.020324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:21.020289    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c" event={"ID":"409da98c-561b-4be9-99cc-6535a33b667f","Type":"ContainerStarted","Data":"ce32c8759c9783d01c021be4a29a8f296ef69e14d047b28e9de3a32b44a5300c"}
Nov 01 06:15:23.031516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:23.031413    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c" event={"ID":"409da98c-561b-4be9-99cc-6535a33b667f","Type":"ContainerStarted","Data":"236a17486411b7ee7a40b2896963d75be367982a3fea0f3bb730519d8e776d52"}
Nov 01 06:15:23.031881 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:23.031520    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:15:23.196665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:23.196596    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c" podStartSLOduration=1.33744981 podStartE2EDuration="3.19658114s" totalImagesPullingTime="1.85913133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:23.194883735 +0000 UTC m=+334.762690566" watchObservedRunningTime="2026-11-01 06:15:23.19658114 +0000 UTC m=+334.764387970"
Nov 01 06:15:32.948315 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:15:32.948276    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:15:44.037002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:15:44.036974    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pwf6c"
Nov 01 06:16:01.061742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.061704    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d475bcc9-z7whx"]
Nov 01 06:16:01.065400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.065376    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.115095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.115043    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.115095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.115087    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7tr\" (UniqueName: \"kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.115331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.115112    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.117697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.117674    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 06:16:01.167421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.167379    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 06:16:01.167421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.167380    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 06:16:01.167421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.167379    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8r64\""
Nov 01 06:16:01.169068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.169046    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-z7whx"]
Nov 01 06:16:01.216272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.216225    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.216272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.216279    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7tr\" (UniqueName: \"kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.216574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.216315    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.219199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.219169    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.219330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.219169    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.272521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.272489    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7tr\" (UniqueName: \"kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr\") pod \"maas-controller-64d475bcc9-z7whx\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") " pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.378838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.378741    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:01.750916 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:01.750832    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-z7whx"]
Nov 01 06:16:01.784025 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:16:01.783993    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3294a44f_c41c_46fa_9844_842944e0ad5e.slice/crio-c175d7d47798564a827f88880174097755f6a0d97e50902602f278e4be67a393 WatchSource:0}: Error finding container c175d7d47798564a827f88880174097755f6a0d97e50902602f278e4be67a393: Status 404 returned error can't find the container with id c175d7d47798564a827f88880174097755f6a0d97e50902602f278e4be67a393
Nov 01 06:16:02.200068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:02.200029    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-z7whx" event={"ID":"3294a44f-c41c-46fa-9844-842944e0ad5e","Type":"ContainerStarted","Data":"c175d7d47798564a827f88880174097755f6a0d97e50902602f278e4be67a393"}
Nov 01 06:16:02.954029 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:16:02.953984    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:16:08.229556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:08.229522    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-z7whx" event={"ID":"3294a44f-c41c-46fa-9844-842944e0ad5e","Type":"ContainerStarted","Data":"f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"}
Nov 01 06:16:08.230021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:08.229755    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:08.345831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:08.345774    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d475bcc9-z7whx" podStartSLOduration=1.14280573 podStartE2EDuration="7.345760208s" totalImagesPullingTime="6.202954478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:08.344176796 +0000 UTC m=+379.911983626" watchObservedRunningTime="2026-11-01 06:16:08.345760208 +0000 UTC m=+379.913567034"
Nov 01 06:16:11.277544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.277506    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"]
Nov 01 06:16:11.282113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.282081    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.345492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.337967    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 06:16:11.345492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.338039    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 06:16:11.345492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.338154    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 06:16:11.386623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.386586    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-c2mdg\""
Nov 01 06:16:11.388326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.388300    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"]
Nov 01 06:16:11.416169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.416125    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q498f\" (UniqueName: \"kubernetes.io/projected/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-kube-api-access-q498f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.416341 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.416180    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.416341 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.416310    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.416553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.416368    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.517723 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.517687    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.517937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.517770    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.517937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.517804    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q498f\" (UniqueName: \"kubernetes.io/projected/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-kube-api-access-q498f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.517937 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:16:11.517811    4468 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 06:16:11.517937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.517842    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.517937 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:16:11.517892    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-proxy-source podName:f3ec2313-e247-4d2a-8d15-465ba35bd9f3 nodeName:}" failed. No retries permitted until 2026-11-01 06:16:12.017871103 +0000 UTC m=+383.585677930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-xg8kb" (UID: "f3ec2313-e247-4d2a-8d15-465ba35bd9f3") : configmap "usage-tenancy-proxy-source" not found
Nov 01 06:16:11.518675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.518651    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.520538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.520518    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:11.579321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:11.579287    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q498f\" (UniqueName: \"kubernetes.io/projected/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-kube-api-access-q498f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:12.022166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:12.022082    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:12.022910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:12.022885    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f3ec2313-e247-4d2a-8d15-465ba35bd9f3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xg8kb\" (UID: \"f3ec2313-e247-4d2a-8d15-465ba35bd9f3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:12.206397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:12.206360    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:12.587705 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:16:12.587676    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ec2313_e247_4d2a_8d15_465ba35bd9f3.slice/crio-81b4a671def26e28d84daec92ed2a3394382b0541abdc07331152c62a9dbc230 WatchSource:0}: Error finding container 81b4a671def26e28d84daec92ed2a3394382b0541abdc07331152c62a9dbc230: Status 404 returned error can't find the container with id 81b4a671def26e28d84daec92ed2a3394382b0541abdc07331152c62a9dbc230
Nov 01 06:16:12.589116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:12.589092    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"]
Nov 01 06:16:13.252579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:13.252541    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb" event={"ID":"f3ec2313-e247-4d2a-8d15-465ba35bd9f3","Type":"ContainerStarted","Data":"81b4a671def26e28d84daec92ed2a3394382b0541abdc07331152c62a9dbc230"}
Nov 01 06:16:19.237381 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:19.236522    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:16:25.367951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.367915    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"]
Nov 01 06:16:25.393029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.392996    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.436974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.436937    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"]
Nov 01 06:16:25.458267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.458225    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-82cfd\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.458485 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.458289    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5mz\" (UniqueName: \"kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz\") pod \"llmisvc-controller-manager-869d5dcc5d-82cfd\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.484945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.484901    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jf8sg\""
Nov 01 06:16:25.484945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.484901    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 01 06:16:25.559631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.559591    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-82cfd\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.559812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.559652    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5mz\" (UniqueName: \"kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz\") pod \"llmisvc-controller-manager-869d5dcc5d-82cfd\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.562336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.562300    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-82cfd\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.617048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.617009    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5mz\" (UniqueName: \"kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz\") pod \"llmisvc-controller-manager-869d5dcc5d-82cfd\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:25.710990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:25.710898    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:26.094070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:26.093992    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"]
Nov 01 06:16:26.322219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:26.322138    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd" event={"ID":"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9","Type":"ContainerStarted","Data":"dd941fb3c0e57134e827487d0f87be1fdc2837911783195d29253cdb8bcbf6b6"}
Nov 01 06:16:26.323763 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:26.323733    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb" event={"ID":"f3ec2313-e247-4d2a-8d15-465ba35bd9f3","Type":"ContainerStarted","Data":"8a15c4dbd489153f3961231b1cf1168519bbb035e7ef039a25ac5db4083b245c"}
Nov 01 06:16:26.324290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:26.323877    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:26.439945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:26.439889    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb" podStartSLOduration=1.9726515610000002 podStartE2EDuration="15.439872353s" totalImagesPullingTime="13.467220792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:26.438859225 +0000 UTC m=+398.006666052" watchObservedRunningTime="2026-11-01 06:16:26.439872353 +0000 UTC m=+398.007679182"
Nov 01 06:16:29.339594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:29.339513    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd" event={"ID":"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9","Type":"ContainerStarted","Data":"7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa"}
Nov 01 06:16:29.339972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:29.339635    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:16:29.454161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:29.454102    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd" podStartSLOduration=1.9704874700000001 podStartE2EDuration="4.454086547s" totalImagesPullingTime="2.483599077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:29.452596471 +0000 UTC m=+401.020403298" watchObservedRunningTime="2026-11-01 06:16:29.454086547 +0000 UTC m=+401.021893436"
Nov 01 06:16:32.948113 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:16:32.948068    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:16:37.334349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:37.334318    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xg8kb"
Nov 01 06:16:47.292552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:16:47.292518    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:17:00.346580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:00.346546    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:17:02.948230 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:17:02.948193    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:17:04.688927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.688888    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"]
Nov 01 06:17:04.691378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.691363    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.793169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.793132    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 06:17:04.793169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.793137    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 06:17:04.793438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.793303    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-8d2rg\""
Nov 01 06:17:04.793438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.793313    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 06:17:04.795228 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.795203    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"]
Nov 01 06:17:04.813018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.812976    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sslgs\" (UniqueName: \"kubernetes.io/projected/a9d7af62-6845-436e-99d2-46058f6ad7c5-kube-api-access-sslgs\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.813205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.813040    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d7af62-6845-436e-99d2-46058f6ad7c5-metrics-certs\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.813205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.813100    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9d7af62-6845-436e-99d2-46058f6ad7c5-manager-config\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.813205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.813134    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9d7af62-6845-436e-99d2-46058f6ad7c5-cert\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.913952 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.913906    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9d7af62-6845-436e-99d2-46058f6ad7c5-manager-config\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.914136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.913967    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9d7af62-6845-436e-99d2-46058f6ad7c5-cert\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.914136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.914000    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sslgs\" (UniqueName: \"kubernetes.io/projected/a9d7af62-6845-436e-99d2-46058f6ad7c5-kube-api-access-sslgs\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.914136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.914045    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d7af62-6845-436e-99d2-46058f6ad7c5-metrics-certs\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.914622 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.914593    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9d7af62-6845-436e-99d2-46058f6ad7c5-manager-config\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.916649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.916618    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9d7af62-6845-436e-99d2-46058f6ad7c5-cert\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.916649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.916645    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d7af62-6845-436e-99d2-46058f6ad7c5-metrics-certs\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:04.970742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:04.970669    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sslgs\" (UniqueName: \"kubernetes.io/projected/a9d7af62-6845-436e-99d2-46058f6ad7c5-kube-api-access-sslgs\") pod \"jobset-controller-manager-6ddb96fb5-b4jlm\" (UID: \"a9d7af62-6845-436e-99d2-46058f6ad7c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:05.002385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:05.002344    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:05.373502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:05.373397    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"]
Nov 01 06:17:05.375774 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:17:05.375740    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d7af62_6845_436e_99d2_46058f6ad7c5.slice/crio-b62a663c70581878bced7aef77fd69528c7ff7084698ea5bfd1f63a15f9462b6 WatchSource:0}: Error finding container b62a663c70581878bced7aef77fd69528c7ff7084698ea5bfd1f63a15f9462b6: Status 404 returned error can't find the container with id b62a663c70581878bced7aef77fd69528c7ff7084698ea5bfd1f63a15f9462b6
Nov 01 06:17:05.512323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:05.512285    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm" event={"ID":"a9d7af62-6845-436e-99d2-46058f6ad7c5","Type":"ContainerStarted","Data":"b62a663c70581878bced7aef77fd69528c7ff7084698ea5bfd1f63a15f9462b6"}
Nov 01 06:17:09.530583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:09.530546    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm" event={"ID":"a9d7af62-6845-436e-99d2-46058f6ad7c5","Type":"ContainerStarted","Data":"fb5c25b6e5d4b69830a326dd663d04db447ba2264f323a2dcc38d7b82d9b660b"}
Nov 01 06:17:09.531063 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:09.530673    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:09.645027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:09.644925    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm" podStartSLOduration=1.663022782 podStartE2EDuration="5.644907662s" totalImagesPullingTime="3.98188488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:09.642660431 +0000 UTC m=+441.210467261" watchObservedRunningTime="2026-11-01 06:17:09.644907662 +0000 UTC m=+441.212714491"
Nov 01 06:17:20.540836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:17:20.540803    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6ddb96fb5-b4jlm"
Nov 01 06:17:32.946551 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:17:32.946494    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:18:02.946320 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:18:02.946237    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:18:15.350285 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.350254    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"]
Nov 01 06:18:15.350835 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.350432    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd" podUID="4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" containerName="manager" containerID="cri-o://7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa" gracePeriod=30
Nov 01 06:18:15.600055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.600035    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:18:15.648052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.647989    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz\" (UniqueName: \"kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz\") pod \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") "
Nov 01 06:18:15.648052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.648039    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert\" (UniqueName: \"kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert\") pod \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\" (UID: \"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9\") "
Nov 01 06:18:15.650299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.650274    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert" pod "4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" (UID: "4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:18:15.650388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.650366    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz" pod "4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" (UID: "4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9"). InnerVolumeSpecName "kube-api-access-hx5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:15.749417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.749385    4468 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:15.749417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.749413    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5mz\" (UniqueName: \"kubernetes.io/projected/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9-kube-api-access-hx5mz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:15.814636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.814604    4468 generic.go:356] "Generic (PLEG): container finished" podID="4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" containerID="7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa" exitCode=0
Nov 01 06:18:15.814808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.814649    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"
Nov 01 06:18:15.814808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.814653    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd" event={"ID":"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9","Type":"ContainerDied","Data":"7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa"}
Nov 01 06:18:15.814808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.814704    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd" event={"ID":"4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9","Type":"ContainerDied","Data":"dd941fb3c0e57134e827487d0f87be1fdc2837911783195d29253cdb8bcbf6b6"}
Nov 01 06:18:15.814808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.814730    4468 scope.go:122] "RemoveContainer" containerID="7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa"
Nov 01 06:18:15.824001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.823984    4468 scope.go:122] "RemoveContainer" containerID="7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa"
Nov 01 06:18:15.824251 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:18:15.824233    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa\": container with ID starting with 7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa not found: ID does not exist" containerID="7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa"
Nov 01 06:18:15.824293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.824260    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa"} err="failed to get container status \"7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa\": rpc error: code = NotFound desc = could not find container \"7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa\": container with ID starting with 7123d1575135358feb3a4dc345920971af14a6edd20ea936f341aac4bb66acaa not found: ID does not exist"
Nov 01 06:18:15.976977 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.976899    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"]
Nov 01 06:18:15.982388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:15.982364    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-82cfd"]
Nov 01 06:18:16.589189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:16.589155    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" path="/var/lib/kubelet/pods/4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9/volumes"
Nov 01 06:18:32.946222 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:18:32.946186    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:18:35.904795 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:35.904765    4468 generic.go:356] "Generic (PLEG): container finished" podID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerID="f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad" exitCode=1
Nov 01 06:18:35.905155 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:35.904830    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-z7whx" event={"ID":"3294a44f-c41c-46fa-9844-842944e0ad5e","Type":"ContainerDied","Data":"f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"}
Nov 01 06:18:35.905205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:35.905166    4468 scope.go:122] "RemoveContainer" containerID="f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"
Nov 01 06:18:36.910778 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:36.910745    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-z7whx" event={"ID":"3294a44f-c41c-46fa-9844-842944e0ad5e","Type":"ContainerStarted","Data":"c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4"}
Nov 01 06:18:36.911149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:36.910964    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:18:47.916918 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:47.916888    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 06:18:58.740289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:18:58.740252    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:19:02.946362 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:19:02.946324    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:19:06.603545 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:06.603511    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:19:10.679184 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.679153    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"]
Nov 01 06:19:10.679594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.679549    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" containerName="manager"
Nov 01 06:19:10.679594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.679560    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" containerName="manager"
Nov 01 06:19:10.679681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.679649    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2e26e2-e9cf-4ce9-8915-a41be3d0e3e9" containerName="manager"
Nov 01 06:19:10.682767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.682750    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.745056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.745027    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.745238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.745102    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.745238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.745159    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.745355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.745274    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.745355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.745319    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrdx\" (UniqueName: \"kubernetes.io/projected/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-kube-api-access-7rrdx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.745458 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.745421    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.784580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.784553    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hczhz\""
Nov 01 06:19:10.784727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.784617    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 06:19:10.784727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.784618    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 06:19:10.784727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.784683    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 06:19:10.786229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.786211    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"]
Nov 01 06:19:10.846726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.846694    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.846893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.846730    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrdx\" (UniqueName: \"kubernetes.io/projected/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-kube-api-access-7rrdx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.846893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.846780    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.846893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.846818    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.846893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.846881    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.847102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.846902    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.847264 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.847242    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.847321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.847266    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.847359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.847336    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.849232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.849205    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.849328 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.849314    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.902838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.902811    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrdx\" (UniqueName: \"kubernetes.io/projected/1cbc55b4-a72b-493e-8cc8-78d4d1456b81-kube-api-access-7rrdx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gh8jf\" (UID: \"1cbc55b4-a72b-493e-8cc8-78d4d1456b81\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:10.995580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:10.995505    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:11.363367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:11.363342    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"]
Nov 01 06:19:11.366034 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:19:11.366001    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbc55b4_a72b_493e_8cc8_78d4d1456b81.slice/crio-0eef451e591806f6032d026f2d8913ace2dc68b005442721dd9c0189b5ac4315 WatchSource:0}: Error finding container 0eef451e591806f6032d026f2d8913ace2dc68b005442721dd9c0189b5ac4315: Status 404 returned error can't find the container with id 0eef451e591806f6032d026f2d8913ace2dc68b005442721dd9c0189b5ac4315
Nov 01 06:19:12.062264 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:12.062228    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf" event={"ID":"1cbc55b4-a72b-493e-8cc8-78d4d1456b81","Type":"ContainerStarted","Data":"0eef451e591806f6032d026f2d8913ace2dc68b005442721dd9c0189b5ac4315"}
Nov 01 06:19:13.068869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:13.068834    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf" event={"ID":"1cbc55b4-a72b-493e-8cc8-78d4d1456b81","Type":"ContainerStarted","Data":"d06ae593b9b74438827fa243d004672d51cb8e9f345d9202d7317771e8af8047"}
Nov 01 06:19:13.182646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:13.182550    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf" podStartSLOduration=1.7078431539999999 podStartE2EDuration="3.182516447s" totalImagesPullingTime="1.474673293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:13.180989705 +0000 UTC m=+564.748796534" watchObservedRunningTime="2026-11-01 06:19:13.182516447 +0000 UTC m=+564.750323276"
Nov 01 06:19:13.996584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:13.996554    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:14.964754 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:14.964722    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:19:18.606296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.606266    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5bs7b"]
Nov 01 06:19:18.609864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.609847    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.711733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.711706    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 06:19:18.713257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.713237    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5bs7b"]
Nov 01 06:19:18.721595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.721574    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.721670 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.721603    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqw2l\" (UniqueName: \"kubernetes.io/projected/7df7649b-cb74-4980-92a2-890009c64658-kube-api-access-wqw2l\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.721775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.721757    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.721810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.721796    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-home\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.721841 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.721822    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df7649b-cb74-4980-92a2-890009c64658-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.721910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.721891    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.822602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.822570    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.822602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.822602    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqw2l\" (UniqueName: \"kubernetes.io/projected/7df7649b-cb74-4980-92a2-890009c64658-kube-api-access-wqw2l\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.822858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.822668    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.822858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.822690    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-home\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.822858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.822710    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df7649b-cb74-4980-92a2-890009c64658-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.822858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.822736    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.823079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.823045    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.823134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.823088    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.823134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.823115    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-home\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.825148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.825125    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df7649b-cb74-4980-92a2-890009c64658-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.825369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.825349    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df7649b-cb74-4980-92a2-890009c64658-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.879843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.879780    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqw2l\" (UniqueName: \"kubernetes.io/projected/7df7649b-cb74-4980-92a2-890009c64658-kube-api-access-wqw2l\") pod \"echo-simulator-kserve-64b6f8f96c-5bs7b\" (UID: \"7df7649b-cb74-4980-92a2-890009c64658\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:18.922502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:18.922453    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:19.297816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:19.291519    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5bs7b"]
Nov 01 06:19:20.108720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:20.108679    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b" event={"ID":"7df7649b-cb74-4980-92a2-890009c64658","Type":"ContainerStarted","Data":"53a51ec66a07690650f22d70d5ab591a159f986e9dfa0d90bba0422d4790d1d0"}
Nov 01 06:19:20.996229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:20.996186    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:21.008521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.008492    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:21.128437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.128404    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gh8jf"
Nov 01 06:19:21.868069 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.868039    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"]
Nov 01 06:19:21.871719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.871697    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:21.927065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.926981    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 06:19:21.928903 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.928867    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"]
Nov 01 06:19:21.952685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.952649    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64l4s\" (UniqueName: \"kubernetes.io/projected/b3a44132-55e7-4bd3-ad44-7c5317b44948-kube-api-access-64l4s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:21.952842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.952699    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a44132-55e7-4bd3-ad44-7c5317b44948-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:21.952842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.952828    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:21.952915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.952862    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:21.952915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.952886    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:21.953004 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:21.952988    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054188    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a44132-55e7-4bd3-ad44-7c5317b44948-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054239    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054361    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054500 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054414    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054565 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054518    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054622    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64l4s\" (UniqueName: \"kubernetes.io/projected/b3a44132-55e7-4bd3-ad44-7c5317b44948-kube-api-access-64l4s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054664    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054682    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.054900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.054879    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.056809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.056792    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a44132-55e7-4bd3-ad44-7c5317b44948-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.056873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.056857    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a44132-55e7-4bd3-ad44-7c5317b44948-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.112074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.112031    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64l4s\" (UniqueName: \"kubernetes.io/projected/b3a44132-55e7-4bd3-ad44-7c5317b44948-kube-api-access-64l4s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx\" (UID: \"b3a44132-55e7-4bd3-ad44-7c5317b44948\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.121216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.121154    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b" event={"ID":"7df7649b-cb74-4980-92a2-890009c64658","Type":"ContainerStarted","Data":"be509fdc9fa3a6bddbc5682235bca266080a68be9ca8e9329d79ba0f2cb7c06b"}
Nov 01 06:19:22.185005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.184974    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:22.238440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.237174    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b" podStartSLOduration=1.748037533 podStartE2EDuration="4.237152106s" totalImagesPullingTime="2.489114573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:22.235154018 +0000 UTC m=+573.802960850" watchObservedRunningTime="2026-11-01 06:19:22.237152106 +0000 UTC m=+573.804958937"
Nov 01 06:19:22.558067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.558047    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"]
Nov 01 06:19:22.559956 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:19:22.559934    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a44132_55e7_4bd3_ad44_7c5317b44948.slice/crio-2531e09b3ab878d2ac265ef4b5091bfbc770fd98075dc78958029c8bde321e2e WatchSource:0}: Error finding container 2531e09b3ab878d2ac265ef4b5091bfbc770fd98075dc78958029c8bde321e2e: Status 404 returned error can't find the container with id 2531e09b3ab878d2ac265ef4b5091bfbc770fd98075dc78958029c8bde321e2e
Nov 01 06:19:22.923495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:22.923403    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:23.127597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:23.127568    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx" event={"ID":"b3a44132-55e7-4bd3-ad44-7c5317b44948","Type":"ContainerStarted","Data":"2531e09b3ab878d2ac265ef4b5091bfbc770fd98075dc78958029c8bde321e2e"}
Nov 01 06:19:24.133503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:24.133446    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx" event={"ID":"b3a44132-55e7-4bd3-ad44-7c5317b44948","Type":"ContainerStarted","Data":"f39b9f632c6dfdd6bb3ae8d7d433ba4664f5fd104a4e71597bbdcadbad0fd257"}
Nov 01 06:19:24.185102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:24.185078    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:24.247209 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:24.247159    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx" podStartSLOduration=2.649142104 podStartE2EDuration="3.247144879s" totalImagesPullingTime="598.002775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:24.245860851 +0000 UTC m=+575.813667679" watchObservedRunningTime="2026-11-01 06:19:24.247144879 +0000 UTC m=+575.814951752"
Nov 01 06:19:25.463407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.463369    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"]
Nov 01 06:19:25.467066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.467048    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.476739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.476704    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:19:25.523576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.523542    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 06:19:25.526051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.526020    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"]
Nov 01 06:19:25.590136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.590102    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkn88\" (UniqueName: \"kubernetes.io/projected/82915854-11e3-466d-a38e-2f235096f61c-kube-api-access-wkn88\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.590306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.590242    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82915854-11e3-466d-a38e-2f235096f61c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.590306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.590279    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.590517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.590483    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.590610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.590548    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.590665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.590614    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692022    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkn88\" (UniqueName: \"kubernetes.io/projected/82915854-11e3-466d-a38e-2f235096f61c-kube-api-access-wkn88\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692083    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82915854-11e3-466d-a38e-2f235096f61c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692107    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692142    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692160    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692179    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692637    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692671    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.692730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.692696    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.694572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.694551    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82915854-11e3-466d-a38e-2f235096f61c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.694682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.694666    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82915854-11e3-466d-a38e-2f235096f61c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.751788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.751724    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkn88\" (UniqueName: \"kubernetes.io/projected/82915854-11e3-466d-a38e-2f235096f61c-kube-api-access-wkn88\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx\" (UID: \"82915854-11e3-466d-a38e-2f235096f61c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:25.783646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:25.783595    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:26.154551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:26.154505    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"]
Nov 01 06:19:26.156852 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:19:26.156826    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82915854_11e3_466d_a38e_2f235096f61c.slice/crio-f0d8f610c3563b0c9d77420cbd3602a6fa33f1205050bd3f006b462590627d3a WatchSource:0}: Error finding container f0d8f610c3563b0c9d77420cbd3602a6fa33f1205050bd3f006b462590627d3a: Status 404 returned error can't find the container with id f0d8f610c3563b0c9d77420cbd3602a6fa33f1205050bd3f006b462590627d3a
Nov 01 06:19:27.148983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:27.148947    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx" event={"ID":"82915854-11e3-466d-a38e-2f235096f61c","Type":"ContainerStarted","Data":"474ca1cfbd4154d5c7e7d8ff41825f665bb3a5b26323b3da1ee13b78b6163fd5"}
Nov 01 06:19:27.149408 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:27.148994    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx" event={"ID":"82915854-11e3-466d-a38e-2f235096f61c","Type":"ContainerStarted","Data":"f0d8f610c3563b0c9d77420cbd3602a6fa33f1205050bd3f006b462590627d3a"}
Nov 01 06:19:27.262452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:27.262386    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx" podStartSLOduration=1.704653991 podStartE2EDuration="2.262373478s" totalImagesPullingTime="557.719487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:27.261684412 +0000 UTC m=+578.829491243" watchObservedRunningTime="2026-11-01 06:19:27.262373478 +0000 UTC m=+578.830180300"
Nov 01 06:19:27.784102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:27.784063    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:28.922683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:28.922647    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:28.934945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:28.934916    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:29.169414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:29.169382    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5bs7b"
Nov 01 06:19:32.186135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:32.186101    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:32.198112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:32.198083    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:32.946993 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:19:32.946953    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:19:33.187743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:33.187715    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx"
Nov 01 06:19:35.784351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:35.784314    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:35.796353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:35.796319    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:36.201739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:36.201714    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx"
Nov 01 06:19:38.161409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:38.161379    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:19:45.139912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.139877    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"]
Nov 01 06:19:45.145672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.145653    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.197425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.197395    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 06:19:45.197597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.197453    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g6wqc\""
Nov 01 06:19:45.248775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.248744    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"]
Nov 01 06:19:45.266074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266035    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266083    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266167    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266221    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266323    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlq2\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266345    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266361    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266409    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.266523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.266447    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367284 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367255    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367298    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367448    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367546    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlq2\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367576    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367591    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367614    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367665    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.367787 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.367696    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.368036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.368009    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.368161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.368016    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.368161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.368075    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.368243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.368166    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.368243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.368180    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.369998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.369980    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.370279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.370262    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.423587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.423526    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.423691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.423581    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlq2\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g2829\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.458558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.458520    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:45.878366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:45.878326    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"]
Nov 01 06:19:45.880583 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:19:45.880553    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf930c3d7_d12f_4acc_8ea8_aa808f252871.slice/crio-ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b WatchSource:0}: Error finding container ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b: Status 404 returned error can't find the container with id ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b
Nov 01 06:19:46.245159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:46.245075    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" event={"ID":"f930c3d7-d12f-4acc-8ea8-aa808f252871","Type":"ContainerStarted","Data":"ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b"}
Nov 01 06:19:49.455636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:49.455603    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:19:49.455889 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:49.455670    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:19:49.455889 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:49.455701    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:19:50.273301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:50.273265    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" event={"ID":"f930c3d7-d12f-4acc-8ea8-aa808f252871","Type":"ContainerStarted","Data":"841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7"}
Nov 01 06:19:50.459383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:50.459346    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:50.460635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:50.460606    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:50.460765 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:50.460666    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:50.489874 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:50.489829    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podStartSLOduration=1.917670395 podStartE2EDuration="5.489816843s" totalImagesPullingTime="3.572146448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:50.439764699 +0000 UTC m=+602.007571518" watchObservedRunningTime="2026-11-01 06:19:50.489816843 +0000 UTC m=+602.057623671"
Nov 01 06:19:51.459178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:51.459147    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:51.459340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:51.459210    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:52.459098 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:52.459063    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:52.459533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:52.459116    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:53.458887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:53.458848    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:53.459074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:53.458912    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:54.459332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:54.459302    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:54.459729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:54.459348    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:55.459880 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:55.459849    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:55.460287 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:55.459906    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:56.459857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:56.459823    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 06:19:56.460060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:56.459874    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 06:19:57.524629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:57.524577    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:57.525092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:57.524968    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:19:57.525703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:19:57.525677    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 06:20:02.946973 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:20:02.946932    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:20:24.735750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.735716    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"]
Nov 01 06:20:24.744403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.744378    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.752787 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.752755    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.752907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.752802    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.752907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.752881    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.753036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.752913    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.753036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.753007    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zw2\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.753175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.753121    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.753229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.753197    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.753307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.753289    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.753403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.753384    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.801278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.801241    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"]
Nov 01 06:20:24.833298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.833261    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-190724-openshift-default-dockercfg-vr7pw\""
Nov 01 06:20:24.854766 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.854740    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.854926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.854774    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.854926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.854794    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.854926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.854899    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zw2\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.854944    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855005    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855097    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855102    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855154    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855235    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855283    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855523    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855630 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855602    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.855745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.855725    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.857339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.857318    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.857674 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.857652    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.911395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.911362    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:24.911611 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:24.911588    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zw2\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:25.057587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:25.057553    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:25.479076 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:25.478993    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"]
Nov 01 06:20:25.482374 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:20:25.482342    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44adfc8e_6de3_4f88_a7bc_23b3042a2320.slice/crio-c0dc649b1952763e93cdbbb98cf14cbb574ed7b33302273f876c63b35f190ea7 WatchSource:0}: Error finding container c0dc649b1952763e93cdbbb98cf14cbb574ed7b33302273f876c63b35f190ea7: Status 404 returned error can't find the container with id c0dc649b1952763e93cdbbb98cf14cbb574ed7b33302273f876c63b35f190ea7
Nov 01 06:20:25.484709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:25.484676    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:20:25.484789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:25.484744    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:20:25.484789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:25.484773    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:20:26.439478 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:26.439429    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" event={"ID":"44adfc8e-6de3-4f88-a7bc-23b3042a2320","Type":"ContainerStarted","Data":"4daa711d74dd81b282b743c2b3a979a874bfa71ed474f4931f6c0ea58e64b725"}
Nov 01 06:20:26.439878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:26.439498    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" event={"ID":"44adfc8e-6de3-4f88-a7bc-23b3042a2320","Type":"ContainerStarted","Data":"c0dc649b1952763e93cdbbb98cf14cbb574ed7b33302273f876c63b35f190ea7"}
Nov 01 06:20:26.605792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:26.605741    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podStartSLOduration=2.60572311 podStartE2EDuration="2.60572311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:26.603454409 +0000 UTC m=+638.171261263" watchObservedRunningTime="2026-11-01 06:20:26.60572311 +0000 UTC m=+638.173529939"
Nov 01 06:20:27.058344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:27.058303    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:27.063513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:27.063485    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:27.443967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:27.443882    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:27.444828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:27.444812    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:20:32.946606 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:20:32.946566    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:20:36.500783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.500751    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"]
Nov 01 06:20:36.504149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.504134    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.561452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.561419    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"]
Nov 01 06:20:36.578106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.578070    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.578283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.578122    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nj28\" (UniqueName: \"kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28\") pod \"payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.608522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.608457    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-190724\""
Nov 01 06:20:36.608693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.608601    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-190724-dockercfg-68w4z\""
Nov 01 06:20:36.659969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.659936    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"]
Nov 01 06:20:36.663403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.663376    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.679297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.679267    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.679432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.679306    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nj28\" (UniqueName: \"kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28\") pod \"payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.679915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.679899    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.712189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.712162    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"]
Nov 01 06:20:36.735755 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.735726    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nj28\" (UniqueName: \"kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28\") pod \"payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.780979 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.780899    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.780979 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.780933    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kf4\" (UniqueName: \"kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4\") pod \"payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.816873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.816846    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:36.882312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.882283    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.882452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.882319    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kf4\" (UniqueName: \"kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4\") pod \"payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.883052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.883026    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.938742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.938713    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kf4\" (UniqueName: \"kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4\") pod \"payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:36.976520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:36.976493    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:37.182567 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:37.182523    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"]
Nov 01 06:20:37.185501 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:20:37.185455    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf297dfd2_be28_4b9d_8ec9_2566568b65c9.slice/crio-246dd23587476fc8094820bf4a3ef573d923d58345c57e26103b9bb54e527d47 WatchSource:0}: Error finding container 246dd23587476fc8094820bf4a3ef573d923d58345c57e26103b9bb54e527d47: Status 404 returned error can't find the container with id 246dd23587476fc8094820bf4a3ef573d923d58345c57e26103b9bb54e527d47
Nov 01 06:20:37.187663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:37.187647    4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:20:37.343129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:37.343102    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"]
Nov 01 06:20:37.345121 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:20:37.345095    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbec8c6_4548_4f51_b683_2864ea8088a5.slice/crio-ecd6f4a41a290dfb933780e4c604d2734c476264875c0ca3646ecb3947b3a413 WatchSource:0}: Error finding container ecd6f4a41a290dfb933780e4c604d2734c476264875c0ca3646ecb3947b3a413: Status 404 returned error can't find the container with id ecd6f4a41a290dfb933780e4c604d2734c476264875c0ca3646ecb3947b3a413
Nov 01 06:20:37.489745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:37.489657    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" event={"ID":"fdbec8c6-4548-4f51-b683-2864ea8088a5","Type":"ContainerStarted","Data":"ecd6f4a41a290dfb933780e4c604d2734c476264875c0ca3646ecb3947b3a413"}
Nov 01 06:20:37.490745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:37.490722    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" event={"ID":"f297dfd2-be28-4b9d-8ec9-2566568b65c9","Type":"ContainerStarted","Data":"246dd23587476fc8094820bf4a3ef573d923d58345c57e26103b9bb54e527d47"}
Nov 01 06:20:39.071320 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.071286    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"]
Nov 01 06:20:39.076373 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.075789    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.178603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.178577    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3ba2f3\""
Nov 01 06:20:39.178603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.178584    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3ba2f3-dockercfg-vl87v\""
Nov 01 06:20:39.181498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.181400    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"]
Nov 01 06:20:39.211188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.211147    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.211370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.211209    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfbt\" (UniqueName: \"kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt\") pod \"payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.312127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.312091    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.312127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.312131    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfbt\" (UniqueName: \"kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt\") pod \"payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.312969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.312941    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.372142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.372069    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfbt\" (UniqueName: \"kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt\") pod \"payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.396843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.396808    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:39.801062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:39.801024    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"]
Nov 01 06:20:40.509123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.509028    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" event={"ID":"fdbec8c6-4548-4f51-b683-2864ea8088a5","Type":"ContainerStarted","Data":"4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410"}
Nov 01 06:20:40.509510 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.509156    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:40.510576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.510556    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" event={"ID":"f297dfd2-be28-4b9d-8ec9-2566568b65c9","Type":"ContainerStarted","Data":"e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9"}
Nov 01 06:20:40.510661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.510594    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:40.511556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.511537    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" event={"ID":"bf29ea2f-d691-4df5-af90-4ff4cd63ba33","Type":"ContainerStarted","Data":"abb956eff951e07ed8ec5bbec4902c6652262ca97a1bbec9e06907cebf6e8aa2"}
Nov 01 06:20:40.622104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.622064    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" podStartSLOduration=1.800547843 podStartE2EDuration="4.622046624s" totalImagesPullingTime="2.821498781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:40.620187907 +0000 UTC m=+652.187994761" watchObservedRunningTime="2026-11-01 06:20:40.622046624 +0000 UTC m=+652.189853454"
Nov 01 06:20:40.735582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:40.735519    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" podStartSLOduration=1.833784968 podStartE2EDuration="4.735497947s" totalImagesPullingTime="2.901712979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:40.733884734 +0000 UTC m=+652.301691562" watchObservedRunningTime="2026-11-01 06:20:40.735497947 +0000 UTC m=+652.303304777"
Nov 01 06:20:41.518315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:41.518274    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" event={"ID":"bf29ea2f-d691-4df5-af90-4ff4cd63ba33","Type":"ContainerStarted","Data":"2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3"}
Nov 01 06:20:41.518803 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:41.518689    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:41.631061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:41.631009    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" podStartSLOduration=1.828905489 podStartE2EDuration="2.630995415s" totalImagesPullingTime="802.089926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:41.629672897 +0000 UTC m=+653.197479730" watchObservedRunningTime="2026-11-01 06:20:41.630995415 +0000 UTC m=+653.198802242"
Nov 01 06:20:42.576936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:42.576898    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:20:50.806577 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:50.806526    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:20:51.521385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:51.521356    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 06:20:51.521575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:51.521430    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 06:20:52.287261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.287227    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"]
Nov 01 06:20:52.291909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.291888    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.344910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.344878    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:20:52.350502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.350448    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.350708 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.350661    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lv66\" (UniqueName: \"kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.350781 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.350719    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.394113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.394082    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:20:52.394113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.394099    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:20:52.394333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.394255    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:20:52.396115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.396095    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"]
Nov 01 06:20:52.451432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.451393    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.451672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.451552    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.451672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.451657    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lv66\" (UniqueName: \"kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.452245 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.452227    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.454119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.454100    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.531230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.531161    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lv66\" (UniqueName: \"kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66\") pod \"maas-api-revoke-sub-507e5841-z2htc\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:52.533897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.533872    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 06:20:52.628763 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:52.628724    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:53.015543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:53.010761    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"]
Nov 01 06:20:53.032432 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:20:53.032396    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184e1f7c_ffd7_4273_841d_b46ba0c3a131.slice/crio-aaa7cd6ebd646074742e6c5faba453c066a389e7c6615ab03312360da2f4e35e WatchSource:0}: Error finding container aaa7cd6ebd646074742e6c5faba453c066a389e7c6615ab03312360da2f4e35e: Status 404 returned error can't find the container with id aaa7cd6ebd646074742e6c5faba453c066a389e7c6615ab03312360da2f4e35e
Nov 01 06:20:53.578406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:53.578368    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc" event={"ID":"184e1f7c-ffd7-4273-841d-b46ba0c3a131","Type":"ContainerStarted","Data":"aaa7cd6ebd646074742e6c5faba453c066a389e7c6615ab03312360da2f4e35e"}
Nov 01 06:20:58.486676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:58.486632    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:20:58.487109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:58.486713    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:20:58.605952 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:58.605915    4468 generic.go:356] "Generic (PLEG): container finished" podID="184e1f7c-ffd7-4273-841d-b46ba0c3a131" containerID="9a82f489fb6e61fb93205e01844f27bb98dd9674a1b100a26f248013dbf7808a" exitCode=0
Nov 01 06:20:58.606147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:58.606011    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc" event={"ID":"184e1f7c-ffd7-4273-841d-b46ba0c3a131","Type":"ContainerDied","Data":"9a82f489fb6e61fb93205e01844f27bb98dd9674a1b100a26f248013dbf7808a"}
Nov 01 06:20:59.773711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.773681    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:20:59.826103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.826066    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca\") pod \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") "
Nov 01 06:20:59.826299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.826150    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66\" (UniqueName: \"kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66\") pod \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") "
Nov 01 06:20:59.826299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.826247    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca\") pod \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\" (UID: \"184e1f7c-ffd7-4273-841d-b46ba0c3a131\") "
Nov 01 06:20:59.826749 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.826717    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca" pod "184e1f7c-ffd7-4273-841d-b46ba0c3a131" (UID: "184e1f7c-ffd7-4273-841d-b46ba0c3a131"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:59.828676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.828648    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca" pod "184e1f7c-ffd7-4273-841d-b46ba0c3a131" (UID: "184e1f7c-ffd7-4273-841d-b46ba0c3a131"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:20:59.828676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.828658    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66" pod "184e1f7c-ffd7-4273-841d-b46ba0c3a131" (UID: "184e1f7c-ffd7-4273-841d-b46ba0c3a131"). InnerVolumeSpecName "kube-api-access-8lv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:59.927174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.927089    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:59.927411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.927388    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e1f7c-ffd7-4273-841d-b46ba0c3a131-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:59.927582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:20:59.927415    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lv66\" (UniqueName: \"kubernetes.io/projected/184e1f7c-ffd7-4273-841d-b46ba0c3a131-kube-api-access-8lv66\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:21:00.621617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.621499    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"
Nov 01 06:21:00.621617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.621522    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc" event={"ID":"184e1f7c-ffd7-4273-841d-b46ba0c3a131","Type":"ContainerDied","Data":"aaa7cd6ebd646074742e6c5faba453c066a389e7c6615ab03312360da2f4e35e"}
Nov 01 06:21:00.621617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.621583    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaa7cd6ebd646074742e6c5faba453c066a389e7c6615ab03312360da2f4e35e"
Nov 01 06:21:00.624189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.623549    4468 generic.go:356] "Generic (PLEG): container finished" podID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerID="4daa711d74dd81b282b743c2b3a979a874bfa71ed474f4931f6c0ea58e64b725" exitCode=0
Nov 01 06:21:00.624189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.623606    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" event={"ID":"44adfc8e-6de3-4f88-a7bc-23b3042a2320","Type":"ContainerDied","Data":"4daa711d74dd81b282b743c2b3a979a874bfa71ed474f4931f6c0ea58e64b725"}
Nov 01 06:21:00.624189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.624054    4468 scope.go:122] "RemoveContainer" containerID="4daa711d74dd81b282b743c2b3a979a874bfa71ed474f4931f6c0ea58e64b725"
Nov 01 06:21:00.625517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.625457    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:00.625598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.625573    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:00.625635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:00.625621    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:01.634399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:01.634357    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" event={"ID":"44adfc8e-6de3-4f88-a7bc-23b3042a2320","Type":"ContainerStarted","Data":"0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8"}
Nov 01 06:21:02.058399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:02.058366    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:21:02.058624 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:02.058599    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 06:21:02.058671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:02.058637    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 06:21:02.952409 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:02.952370    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:21:03.058429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:03.058391    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 06:21:03.058670 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:03.058455    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 06:21:04.058252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:04.058216    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 06:21:04.058701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:04.058268    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 06:21:05.057926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:05.057889    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 06:21:05.058147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:05.057939    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 06:21:06.058220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:06.058176    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 06:21:06.058704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:06.058268    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 06:21:07.061909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:07.061879    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:21:07.062367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:07.062175    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:21:07.062871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:07.062852    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:21:11.056057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.056025    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"]
Nov 01 06:21:11.056503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.056488    4468 cpu_manager.go:414] "containerMap: removing container" podUID="184e1f7c-ffd7-4273-841d-b46ba0c3a131" containerName="revoke-keys"
Nov 01 06:21:11.056546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.056507    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e1f7c-ffd7-4273-841d-b46ba0c3a131" containerName="revoke-keys"
Nov 01 06:21:11.056637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.056612    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e1f7c-ffd7-4273-841d-b46ba0c3a131" containerName="revoke-keys"
Nov 01 06:21:11.061930 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.061912    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145099    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145145    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145242    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6lp\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145285    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145382    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145402    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145428    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145443    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.145569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.145553    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.165029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.164996    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"]
Nov 01 06:21:11.167516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.167493    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-85584b-openshift-default-dockercfg-ldlm7\""
Nov 01 06:21:11.247028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.246986    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247032    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247051    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247092    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6lp\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247124    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247207    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247233    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247290    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247316    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247593    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247641    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247710    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247821    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.247872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.247841    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.249789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.249767    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.249869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.249851    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.304037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.304008    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.304430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.304410    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6lp\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp\") pod \"e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") " pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.374085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.373994    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:11.796180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.796152    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"]
Nov 01 06:21:11.798221 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:21:11.798194    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c31f27_e3c6_40e5_895c_9cae71d50475.slice/crio-313c541c920dac1e74325e598fd0f7da73465f69e588688c7fcac98e8b122374 WatchSource:0}: Error finding container 313c541c920dac1e74325e598fd0f7da73465f69e588688c7fcac98e8b122374: Status 404 returned error can't find the container with id 313c541c920dac1e74325e598fd0f7da73465f69e588688c7fcac98e8b122374
Nov 01 06:21:11.800359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.800329    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:11.800408 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.800395    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:11.800444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:11.800425    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:12.696345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:12.696308    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" event={"ID":"b4c31f27-e3c6-40e5-895c-9cae71d50475","Type":"ContainerStarted","Data":"85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"}
Nov 01 06:21:12.696747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:12.696361    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" event={"ID":"b4c31f27-e3c6-40e5-895c-9cae71d50475","Type":"ContainerStarted","Data":"313c541c920dac1e74325e598fd0f7da73465f69e588688c7fcac98e8b122374"}
Nov 01 06:21:12.909535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:12.909455    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podStartSLOduration=1.909440788 podStartE2EDuration="1.909440788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:12.859790934 +0000 UTC m=+684.427597760" watchObservedRunningTime="2026-11-01 06:21:12.909440788 +0000 UTC m=+684.477247615"
Nov 01 06:21:13.374791 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:13.374756    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:13.379983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:13.379958    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:13.701161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:13.701077    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:13.702108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:13.702090    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:14.413963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:14.413928    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:21:30.321600 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.321568    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"]
Nov 01 06:21:30.325829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.325810    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.371113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.371091    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"]
Nov 01 06:21:30.374739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.374723    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.426936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.426909    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"]
Nov 01 06:21:30.445363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445335    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445374    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445408    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445538    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445574    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445663    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445684    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445724    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.445808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.445788    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbxt\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.483198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.483172    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"]
Nov 01 06:21:30.546644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546621    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbxt\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.546759 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546656    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.546759 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546698    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.546759 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546725    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxft\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.546759 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546743    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.546954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546825    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.546954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546870    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.546954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546890    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.546974    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547020    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547054    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547163    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547196    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547226    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547258    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547257    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547299    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547311    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547524    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.547718 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547605    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547718 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547626    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547718 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547629    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.547968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.547948    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.549250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.549223    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.549491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.549449    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.602796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.602730    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbxt\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.603123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.603106    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.639120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.639096    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:30.648356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648340    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648378    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxft\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648419    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648440    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648456    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648507    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648538    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648634    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.648773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.648675    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.649080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.649055    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.649196 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.649102    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.649196 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.649135    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.649383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.649362    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.649599 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.649538    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.651196 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.651173    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.651334 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.651195    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.704306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.704288    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.704525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.704509    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxft\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft\") pod \"maas-default-gateway-openshift-default-7d9897b54c-g6cln\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:30.986696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:30.986634    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:32.947418 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:32.947377    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:21:33.234619 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.234584    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:33.234619 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e" Netns:"/var/run/netns/1ef4a2c8-00c7-405a-a254-e5072777328e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:33.234619 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:33.234619 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.234646    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e" Netns:"/var/run/netns/1ef4a2c8-00c7-405a-a254-e5072777328e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.234665    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e" Netns:"/var/run/netns/1ef4a2c8-00c7-405a-a254-e5072777328e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:33.234795 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.234732    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress(83242ea6-ab3a-4ac0-a37e-012b6c0b4265)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress(83242ea6-ab3a-4ac0-a37e-012b6c0b4265)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e\\\" Netns:\\\"/var/run/netns/1ef4a2c8-00c7-405a-a254-e5072777328e\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=8b278b9bce9ae0ab27a05466bb8b9ce1d471bb4c55e988595a83439b3b82cf0e;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265"
Nov 01 06:21:33.577432 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.577399    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:33.577432 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed" Netns:"/var/run/netns/0302f7d5-0f56-417e-b291-b36e61be63bc" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:33.577432 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:33.577432 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.577454    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed" Netns:"/var/run/netns/0302f7d5-0f56-417e-b291-b36e61be63bc" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.577490    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed" Netns:"/var/run/netns/0302f7d5-0f56-417e-b291-b36e61be63bc" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:33.577633 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:33.577575    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress(8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress(8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed\\\" Netns:\\\"/var/run/netns/0302f7d5-0f56-417e-b291-b36e61be63bc\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=c6dfe699a2568e3f1461c011f2e1a8a7907be568bffa813a45776186e9c913ed;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"
Nov 01 06:21:33.800361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:33.800326    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:33.800557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:33.800328    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:33.800658 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:33.800647    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:33.800762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:33.800751    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:36.396093 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396061    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:36.396093 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029" Netns:"/var/run/netns/af072dc1-7674-4037-9207-828ff7ec00f3" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:36.396093 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:36.396093 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396122    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029" Netns:"/var/run/netns/af072dc1-7674-4037-9207-828ff7ec00f3" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396147    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029" Netns:"/var/run/netns/af072dc1-7674-4037-9207-828ff7ec00f3" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396127    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04" Netns:"/var/run/netns/922c9c66-1f17-4981-b566-7ec272a0c922" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396228    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04" Netns:"/var/run/netns/922c9c66-1f17-4981-b566-7ec272a0c922" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396247    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04" Netns:"/var/run/netns/922c9c66-1f17-4981-b566-7ec272a0c922" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396227    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress(8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress(8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029\\\" Netns:\\\"/var/run/netns/af072dc1-7674-4037-9207-828ff7ec00f3\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=3e138a7829c55026afd9aae87527a835bfbd9d12ac0911101a881ef254aaa029;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"
Nov 01 06:21:36.396625 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:36.396302    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress(83242ea6-ab3a-4ac0-a37e-012b6c0b4265)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress(83242ea6-ab3a-4ac0-a37e-012b6c0b4265)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04\\\" Netns:\\\"/var/run/netns/922c9c66-1f17-4981-b566-7ec272a0c922\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=1f9f369da12d30ceb8322c3d6d4efcd380f33f92682339c34ceeb0f7b5e9be04;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265"
Nov 01 06:21:39.315511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:39.315455    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:21:45.330542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.330505    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"]
Nov 01 06:21:45.335430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.335407    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.413744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.413709    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.413926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.413764    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.413926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.413827    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.413926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.413893    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.414036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.414001    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.414091 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.414075    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.414126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.414114    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.414222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.414208    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9729l\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.414279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.414258    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.439191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.439161    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"]
Nov 01 06:21:45.515833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.515801    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.515860    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.515893    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.515929    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.515955    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.515986    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9729l\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516017    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516066    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516112    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516262    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516609    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516788    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.516983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.516962    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.517042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.517012    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.519272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.519244    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.519363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.519289    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.572956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.572922    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.573184 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.573166    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9729l\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l\") pod \"maas-default-gateway-openshift-default-7d9897b54c-6zskg\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:45.656605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:45.656519    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:48.249751 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:48.249714    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:48.249751 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38" Netns:"/var/run/netns/e80589aa-88ac-4da2-91e6-64e2ad82cea5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s": context deadline exceeded
Nov 01 06:21:48.249751 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:48.249751 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:48.249769    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38" Netns:"/var/run/netns/e80589aa-88ac-4da2-91e6-64e2ad82cea5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s": context deadline exceeded
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:48.249786    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38" Netns:"/var/run/netns/e80589aa-88ac-4da2-91e6-64e2ad82cea5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s": context deadline exceeded
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:48.250182 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:48.249870    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress(db5ba475-fdb0-417a-8799-d8448ad561dc)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress(db5ba475-fdb0-417a-8799-d8448ad561dc)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38\\\" Netns:\\\"/var/run/netns/e80589aa-88ac-4da2-91e6-64e2ad82cea5\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c57356a18c1053d701d00be65e0f2ae26051049532904f11903b70f0858cee38;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc"
Nov 01 06:21:48.585520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.585495    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:48.585520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.585515    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:48.585754 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.585740    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:48.585842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.585826    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:48.880260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.880227    4468 generic.go:356] "Generic (PLEG): container finished" podID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerID="85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549" exitCode=0
Nov 01 06:21:48.880419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.880303    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" event={"ID":"b4c31f27-e3c6-40e5-895c-9cae71d50475","Type":"ContainerDied","Data":"85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"}
Nov 01 06:21:48.880419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.880318    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:48.880627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.880615    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:48.880744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.880729    4468 scope.go:122] "RemoveContainer" containerID="85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"
Nov 01 06:21:48.881877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.881846    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:48.881981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.881924    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:48.881981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:48.881956    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:21:49.892854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:49.892818    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" event={"ID":"b4c31f27-e3c6-40e5-895c-9cae71d50475","Type":"ContainerStarted","Data":"27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91"}
Nov 01 06:21:50.374542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:50.374504    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:50.374739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:50.374716    4468 patch_prober.go:29] interesting pod/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 06:21:50.374793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:50.374773    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184279    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0" Netns:"/var/run/netns/f17d5b3d-3d2a-42f2-8840-4c93c4743b21" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184327    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0" Netns:"/var/run/netns/f17d5b3d-3d2a-42f2-8840-4c93c4743b21" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184344    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0" Netns:"/var/run/netns/f17d5b3d-3d2a-42f2-8840-4c93c4743b21" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:21:51.184578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184423    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress(8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress(8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-g6cln_openshift-ingress_8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a_0(1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-g6cln to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0\\\" Netns:\\\"/var/run/netns/f17d5b3d-3d2a-42f2-8840-4c93c4743b21\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-g6cln;K8S_POD_INFRA_CONTAINER_ID=1c85a7f05e6d6c55dc3cbd3f1033c0606f1590430eda2e01b0c106861eb345a0;K8S_POD_UID=8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-g6cln?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184746    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99" Netns:"/var/run/netns/ea3e98a6-e61c-449f-a33c-bc533b332152" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184794    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99" Netns:"/var/run/netns/ea3e98a6-e61c-449f-a33c-bc533b332152" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184810    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99" Netns:"/var/run/netns/ea3e98a6-e61c-449f-a33c-bc533b332152" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:21:51.185550 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.184858    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress(83242ea6-ab3a-4ac0-a37e-012b6c0b4265)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress(83242ea6-ab3a-4ac0-a37e-012b6c0b4265)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-nrzv7_openshift-ingress_83242ea6-ab3a-4ac0-a37e-012b6c0b4265_0(015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-nrzv7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99\\\" Netns:\\\"/var/run/netns/ea3e98a6-e61c-449f-a33c-bc533b332152\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-nrzv7;K8S_POD_INFRA_CONTAINER_ID=015cfd17b3b4dc2bb859b9bbafc643986decccdda71085b005711f5284910c99;K8S_POD_UID=83242ea6-ab3a-4ac0-a37e-012b6c0b4265\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7/83242ea6-ab3a-4ac0-a37e-012b6c0b4265]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-nrzv7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265"
Nov 01 06:21:51.374838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:51.374803    4468 patch_prober.go:29] interesting pod/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 06:21:51.375014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:51.374854    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 06:21:51.464538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:51.464508    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:21:51.477391 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.477363    4468 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:21:51.477391 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf" Netns:"/var/run/netns/bd8d0746-44d5-432f-b6e5-f4d645b557e6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.477391 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.477391 ip-10-0-138-21 kubenswrapper[4468]:  >
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.477409    4468 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf" Netns:"/var/run/netns/bd8d0746-44d5-432f-b6e5-f4d645b557e6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.477426    4468 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf" Netns:"/var/run/netns/bd8d0746-44d5-432f-b6e5-f4d645b557e6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc" Path:"" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get "https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s": context deadline exceeded
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]:  > pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:21:51.477560 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:21:51.477519    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress(db5ba475-fdb0-417a-8799-d8448ad561dc)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress(db5ba475-fdb0-417a-8799-d8448ad561dc)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-default-gateway-openshift-default-7d9897b54c-6zskg_openshift-ingress_db5ba475-fdb0-417a-8799-d8448ad561dc_0(c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf): error adding pod openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-6zskg to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf\\\" Netns:\\\"/var/run/netns/bd8d0746-44d5-432f-b6e5-f4d645b557e6\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=maas-default-gateway-openshift-default-7d9897b54c-6zskg;K8S_POD_INFRA_CONTAINER_ID=c2233bf2595d00f82f1b6731d8ff1244b240ca9cd07cdea434f3f8656fcfeeaf;K8S_POD_UID=db5ba475-fdb0-417a-8799-d8448ad561dc\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg] networking: Multus: [openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg/db5ba475-fdb0-417a-8799-d8448ad561dc]: error waiting for pod: Get \\\"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/maas-default-gateway-openshift-default-7d9897b54c-6zskg?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc"
Nov 01 06:21:52.374539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:52.374501    4468 patch_prober.go:29] interesting pod/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 06:21:52.374925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:52.374587    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 06:21:53.375250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:53.375207    4468 patch_prober.go:29] interesting pod/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 06:21:53.375685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:53.375271    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 06:21:54.375119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:54.375082    4468 patch_prober.go:29] interesting pod/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 06:21:54.375312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:54.375138    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 06:21:55.378868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:55.378833    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:55.379348 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:55.379229    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:21:55.379955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:21:55.379938    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:22:02.946673 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:02.946636    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:22:04.582885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:04.582856    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:22:04.582885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:04.582881    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:22:04.583418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:04.583240    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:22:04.583456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:04.583422    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:22:05.064060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.064034    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"]
Nov 01 06:22:05.065636 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:05.065614    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5ba475_fdb0_417a_8799_d8448ad561dc.slice/crio-945628bcd9370a10571911985860c8d8865af167a72fff07281f4a16fdaa09a6 WatchSource:0}: Error finding container 945628bcd9370a10571911985860c8d8865af167a72fff07281f4a16fdaa09a6: Status 404 returned error can't find the container with id 945628bcd9370a10571911985860c8d8865af167a72fff07281f4a16fdaa09a6
Nov 01 06:22:05.067975 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.067945    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:05.068029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.068009    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:05.068064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.068040    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:05.113489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.113452    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"]
Nov 01 06:22:05.114952 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:05.114909    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83242ea6_ab3a_4ac0_a37e_012b6c0b4265.slice/crio-59ad34e13e2c5e17dff1a8983f586c0cee00fe3a128d84b9706d82a77bf55b0f WatchSource:0}: Error finding container 59ad34e13e2c5e17dff1a8983f586c0cee00fe3a128d84b9706d82a77bf55b0f: Status 404 returned error can't find the container with id 59ad34e13e2c5e17dff1a8983f586c0cee00fe3a128d84b9706d82a77bf55b0f
Nov 01 06:22:05.117264 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.117233    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:05.117378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.117306    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:05.117378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.117352    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:05.996428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.996391    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" event={"ID":"db5ba475-fdb0-417a-8799-d8448ad561dc","Type":"ContainerStarted","Data":"8ad0a1f14db500d811011ff4f2f4c7b4b908d55a6273e4caf1d4be747640de1f"}
Nov 01 06:22:05.997153 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.997020    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" event={"ID":"db5ba475-fdb0-417a-8799-d8448ad561dc","Type":"ContainerStarted","Data":"945628bcd9370a10571911985860c8d8865af167a72fff07281f4a16fdaa09a6"}
Nov 01 06:22:05.998296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.998274    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" event={"ID":"83242ea6-ab3a-4ac0-a37e-012b6c0b4265","Type":"ContainerStarted","Data":"deb420d2b7bc4b4b9c08408a091aa49c1dfa9bdfb7c182da90daeeb77bcb30cc"}
Nov 01 06:22:05.998375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:05.998314    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" event={"ID":"83242ea6-ab3a-4ac0-a37e-012b6c0b4265","Type":"ContainerStarted","Data":"59ad34e13e2c5e17dff1a8983f586c0cee00fe3a128d84b9706d82a77bf55b0f"}
Nov 01 06:22:06.229494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.226312    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podStartSLOduration=21.226291308 podStartE2EDuration="21.226291308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:06.166339553 +0000 UTC m=+737.734146390" watchObservedRunningTime="2026-11-01 06:22:06.226291308 +0000 UTC m=+737.794098137"
Nov 01 06:22:06.288331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.288201    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podStartSLOduration=36.288178958 podStartE2EDuration="36.288178958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:06.286645499 +0000 UTC m=+737.854452337" watchObservedRunningTime="2026-11-01 06:22:06.288178958 +0000 UTC m=+737.855985788"
Nov 01 06:22:06.583515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.583457    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:22:06.583750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.583737    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:22:06.643521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.642858    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:22:06.659937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.657886    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:22:06.659937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.658717    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:06.659937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.658768    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:06.659937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.659572    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:06.659937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:06.659620    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:07.064665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.064429    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"]
Nov 01 06:22:07.076058 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:07.076022    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1fdc09_8d2b_4c4e_bb72_3c4e15342f8a.slice/crio-560e20f3d46f25ede61331605ea519cb1d1afab2e6754b4e650aec55963c4094 WatchSource:0}: Error finding container 560e20f3d46f25ede61331605ea519cb1d1afab2e6754b4e650aec55963c4094: Status 404 returned error can't find the container with id 560e20f3d46f25ede61331605ea519cb1d1afab2e6754b4e650aec55963c4094
Nov 01 06:22:07.080610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.080574    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:07.080840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.080817    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:07.080995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.080979    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:22:07.644492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.639855    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:07.644492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.639917    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:07.657503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.656930    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:07.657503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:07.657001    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:08.046586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.042694    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" event={"ID":"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a","Type":"ContainerStarted","Data":"706a069381ba00d1308450cc9f192f6f8dcaccb0fb9b907b402bd8b64001d4e7"}
Nov 01 06:22:08.046586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.043412    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" event={"ID":"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a","Type":"ContainerStarted","Data":"560e20f3d46f25ede61331605ea519cb1d1afab2e6754b4e650aec55963c4094"}
Nov 01 06:22:08.184729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.184378    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podStartSLOduration=38.184358099 podStartE2EDuration="38.184358099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:08.166423623 +0000 UTC m=+739.734230453" watchObservedRunningTime="2026-11-01 06:22:08.184358099 +0000 UTC m=+739.752164930"
Nov 01 06:22:08.642492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.639883    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:08.642492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.639947    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:08.657895 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.657861    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:08.658182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.658154    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:08.988192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.988103    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:22:08.988192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.988115    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:08.988391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:08.988191    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:09.652491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:09.651856    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:09.652491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:09.651918    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:09.679509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:09.678849    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:09.679509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:09.678908    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:09.987956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:09.987855    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:09.987956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:09.987917    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:10.643491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:10.643016    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:10.643491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:10.643095    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:10.663491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:10.658890    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:10.663491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:10.658951    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:10.988127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:10.988037    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:10.988127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:10.988091    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:11.640263 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:11.640226    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:11.640480 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:11.640288    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:11.657374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:11.657344    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:11.657514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:11.657390    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:11.987514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:11.987404    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:11.987971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:11.987515    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:12.640286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:12.640239    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-nrzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 01 06:22:12.640520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:12.640297    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 01 06:22:12.657754 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:12.657723    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-6zskg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 06:22:12.657906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:12.657775    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 06:22:12.988120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:12.988015    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:12.988120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:12.988101    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:13.783771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.783737    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:22:13.784324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.784288    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:22:13.785359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.785336    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:22:13.986329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.986281    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:22:13.986771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.986747    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:22:13.987070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.986976    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:13.987070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.987039    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:13.988908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:13.988887    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:22:14.987944 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:14.987909    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g6cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:22:14.988102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:14.987964    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:22:15.987748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:15.987682    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:22:15.991834 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:15.991812    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:22:16.116091 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:16.116066    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:22:23.296808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:23.296776    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:22:32.946402 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:32.946329    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:22:42.012007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.011974    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"]
Nov 01 06:22:42.021750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.021721    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.073893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.073864    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"]
Nov 01 06:22:42.120306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.120271    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"openshift-service-ca.crt\""
Nov 01 06:22:42.120306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.120288    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"kube-root-ca.crt\""
Nov 01 06:22:42.120520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.120336    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"default-dockercfg-hl7g2\""
Nov 01 06:22:42.123193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.123177    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"premium-w2-8ff537-kserve-self-signed-certs\""
Nov 01 06:22:42.173041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.173008    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tn2\" (UniqueName: \"kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.173225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.173175    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.173225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.173203    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.173347 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.173259    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.173347 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.173287    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.173347 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.173310    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.274595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.274508    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.274595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.274552    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.274595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.274596    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.274830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.274613    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.274830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.274629    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.274830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.274644    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tn2\" (UniqueName: \"kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.275095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.275063    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.275213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.275120    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.275213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.275173    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.276984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.276966    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.277141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.277126    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.330817 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.330789    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tn2\" (UniqueName: \"kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2\") pod \"premium-w2-8ff537-kserve-68c6579b79-wrrvg\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") " pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.332621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.332603    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:42.731794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:42.731765    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"]
Nov 01 06:22:42.734120 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:42.734091    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e555b8c_c82b_4d6f_98ff_cc081af6454f.slice/crio-f51a5d2a6822c17bd3cc3c123dfc3ca897f30b1fe26aae6c9b09791618666ab7 WatchSource:0}: Error finding container f51a5d2a6822c17bd3cc3c123dfc3ca897f30b1fe26aae6c9b09791618666ab7: Status 404 returned error can't find the container with id f51a5d2a6822c17bd3cc3c123dfc3ca897f30b1fe26aae6c9b09791618666ab7
Nov 01 06:22:43.257412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:43.257377    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" event={"ID":"4e555b8c-c82b-4d6f-98ff-cc081af6454f","Type":"ContainerStarted","Data":"f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f"}
Nov 01 06:22:43.257828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:43.257421    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" event={"ID":"4e555b8c-c82b-4d6f-98ff-cc081af6454f","Type":"ContainerStarted","Data":"f51a5d2a6822c17bd3cc3c123dfc3ca897f30b1fe26aae6c9b09791618666ab7"}
Nov 01 06:22:43.332967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:43.332939    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:43.370573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:43.370524    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podStartSLOduration=2.370503941 podStartE2EDuration="2.370503941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:43.368950644 +0000 UTC m=+774.936757508" watchObservedRunningTime="2026-11-01 06:22:43.370503941 +0000 UTC m=+774.938310770"
Nov 01 06:22:45.939748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:45.939714    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:22:51.451204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.451168    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7689-6a6333"]
Nov 01 06:22:51.455036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.455018    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:51.507802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.507772    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\""
Nov 01 06:22:51.510210 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.510185    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-6a6333"]
Nov 01 06:22:51.568145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.568112    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vn9\" (UniqueName: \"kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9\") pod \"test-curl-7689-6a6333\" (UID: \"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8\") " pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:51.668711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.668682    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vn9\" (UniqueName: \"kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9\") pod \"test-curl-7689-6a6333\" (UID: \"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8\") " pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:51.724729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.724650    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vn9\" (UniqueName: \"kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9\") pod \"test-curl-7689-6a6333\" (UID: \"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8\") " pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:51.766955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:51.766934    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:52.133938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:52.133916    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-6a6333"]
Nov 01 06:22:52.136227 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:52.136201    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50a7805c_f886_4e5c_aa5f_f9a64cc8c1a8.slice/crio-3d2fab7ed99a49c5544e44b332a4a58a35bfbb701a6cdc6a279b9aa70494e874 WatchSource:0}: Error finding container 3d2fab7ed99a49c5544e44b332a4a58a35bfbb701a6cdc6a279b9aa70494e874: Status 404 returned error can't find the container with id 3d2fab7ed99a49c5544e44b332a4a58a35bfbb701a6cdc6a279b9aa70494e874
Nov 01 06:22:52.304225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:52.304194    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-6a6333" event={"ID":"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8","Type":"ContainerStarted","Data":"3d2fab7ed99a49c5544e44b332a4a58a35bfbb701a6cdc6a279b9aa70494e874"}
Nov 01 06:22:52.333301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:52.333275    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:52.345593 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:52.345573    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:53.324548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:53.324518    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 06:22:55.323924 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:55.323895    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-6a6333" event={"ID":"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8","Type":"ContainerStarted","Data":"2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1"}
Nov 01 06:22:55.434730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:55.434675    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7689-6a6333" podStartSLOduration=1.8475208749999998 podStartE2EDuration="4.434662256s" totalImagesPullingTime="2.587141381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:55.434145662 +0000 UTC m=+787.001952496" watchObservedRunningTime="2026-11-01 06:22:55.434662256 +0000 UTC m=+787.002469085"
Nov 01 06:22:56.298298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.298263    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"]
Nov 01 06:22:56.302072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.302052    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.405307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.405260    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:22:56.405307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.405267    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:22:56.405307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.405311    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:22:56.405812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.405372    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:22:56.407801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.407777    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"]
Nov 01 06:22:56.420423 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.420402    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.420534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.420430    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.420534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.420454    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd82\" (UniqueName: \"kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.521689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.521660    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.521813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.521694    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.521813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.521717    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd82\" (UniqueName: \"kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.522388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.522367    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.524088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.524070    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.578926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.578857    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd82\" (UniqueName: \"kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82\") pod \"maas-api-revoke-sub-ca54dda5-6csmc\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.616836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.616813    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:56.799705 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.799674    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7689-6a6333"]
Nov 01 06:22:56.804664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.804628    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7689-6a6333"]
Nov 01 06:22:56.956098 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.956023    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7689-899601"]
Nov 01 06:22:56.956551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.956537    4468 cpu_manager.go:414] "containerMap: removing container" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" containerName="test-curl-7689-6a6333"
Nov 01 06:22:56.956609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.956553    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" containerName="test-curl-7689-6a6333"
Nov 01 06:22:56.956643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.956627    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" containerName="test-curl-7689-6a6333"
Nov 01 06:22:56.960991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.960970    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-899601"
Nov 01 06:22:56.991058 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:56.991034    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"]
Nov 01 06:22:56.993069 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:56.993045    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0a8b0d_b7a8_4af8_b3e7_3c1cccc9c3f9.slice/crio-c703a073fd337879b33b4ad041d77553f77c16b4c5b1282c1327d3fab4fe98c6 WatchSource:0}: Error finding container c703a073fd337879b33b4ad041d77553f77c16b4c5b1282c1327d3fab4fe98c6: Status 404 returned error can't find the container with id c703a073fd337879b33b4ad041d77553f77c16b4c5b1282c1327d3fab4fe98c6
Nov 01 06:22:57.013577 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.013144    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-899601"]
Nov 01 06:22:57.026796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.026770    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njb9f\" (UniqueName: \"kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f\") pod \"test-curl-7689-899601\" (UID: \"db6f4565-1da1-4c79-83b9-53beab67b8a2\") " pod="opendatahub/test-curl-7689-899601"
Nov 01 06:22:57.128419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.128392    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njb9f\" (UniqueName: \"kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f\") pod \"test-curl-7689-899601\" (UID: \"db6f4565-1da1-4c79-83b9-53beab67b8a2\") " pod="opendatahub/test-curl-7689-899601"
Nov 01 06:22:57.184698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.184671    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njb9f\" (UniqueName: \"kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f\") pod \"test-curl-7689-899601\" (UID: \"db6f4565-1da1-4c79-83b9-53beab67b8a2\") " pod="opendatahub/test-curl-7689-899601"
Nov 01 06:22:57.273489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.273426    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-899601"
Nov 01 06:22:57.346171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.346134    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7689-6a6333" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" containerName="test-curl-7689-6a6333" containerID="cri-o://2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1" gracePeriod=2
Nov 01 06:22:57.346682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.346613    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc" event={"ID":"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9","Type":"ContainerStarted","Data":"f77a99f8c1cc541045041a3ddc594c12468b7426c16503b040090ea1bf2f36a4"}
Nov 01 06:22:57.346682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.346665    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc" event={"ID":"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9","Type":"ContainerStarted","Data":"c703a073fd337879b33b4ad041d77553f77c16b4c5b1282c1327d3fab4fe98c6"}
Nov 01 06:22:57.397047 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:57.397014    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:57.510531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.510424    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc" podStartSLOduration=1.510410064 podStartE2EDuration="1.510410064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:57.507948773 +0000 UTC m=+789.075755637" watchObservedRunningTime="2026-11-01 06:22:57.510410064 +0000 UTC m=+789.078216893"
Nov 01 06:22:57.558434 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:57.558393    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:57.643254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:57.643232    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7689-899601"]
Nov 01 06:22:57.645227 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:22:57.645198    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb6f4565_1da1_4c79_83b9_53beab67b8a2.slice/crio-bbf75375c0954d75c1ed99a7b76c72f0b10b9fba105907df809aa9b45bcdd051 WatchSource:0}: Error finding container bbf75375c0954d75c1ed99a7b76c72f0b10b9fba105907df809aa9b45bcdd051: Status 404 returned error can't find the container with id bbf75375c0954d75c1ed99a7b76c72f0b10b9fba105907df809aa9b45bcdd051
Nov 01 06:22:58.353162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:58.353132    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-899601" event={"ID":"db6f4565-1da1-4c79-83b9-53beab67b8a2","Type":"ContainerStarted","Data":"6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b"}
Nov 01 06:22:58.353363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:58.353177    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7689-899601" event={"ID":"db6f4565-1da1-4c79-83b9-53beab67b8a2","Type":"ContainerStarted","Data":"bbf75375c0954d75c1ed99a7b76c72f0b10b9fba105907df809aa9b45bcdd051"}
Nov 01 06:22:58.354700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:58.354663    4468 generic.go:356] "Generic (PLEG): container finished" podID="cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" containerID="f77a99f8c1cc541045041a3ddc594c12468b7426c16503b040090ea1bf2f36a4" exitCode=0
Nov 01 06:22:58.354808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:58.354701    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc" event={"ID":"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9","Type":"ContainerDied","Data":"f77a99f8c1cc541045041a3ddc594c12468b7426c16503b040090ea1bf2f36a4"}
Nov 01 06:22:58.403949 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:58.403909    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:58.564322 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:58.564284    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:58.677939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:58.677848    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7689-899601" podStartSLOduration=2.6778354650000002 podStartE2EDuration="2.677835465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:58.513703152 +0000 UTC m=+790.081509983" watchObservedRunningTime="2026-11-01 06:22:58.677835465 +0000 UTC m=+790.245642293"
Nov 01 06:22:59.500629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.500595    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:22:59.553513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.553454    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca\") pod \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") "
Nov 01 06:22:59.553653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.553554    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca\") pod \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") "
Nov 01 06:22:59.553653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.553640    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82\" (UniqueName: \"kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82\") pod \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\" (UID: \"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9\") "
Nov 01 06:22:59.553861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.553838    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca" pod "cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" (UID: "cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:22:59.554000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.553983    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:22:59.555551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.555527    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca" pod "cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" (UID: "cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:22:59.555656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.555600    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82" pod "cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" (UID: "cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9"). InnerVolumeSpecName "kube-api-access-gwd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:22:59.575855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.575833    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:59.654743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.654673    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9\" (UniqueName: \"kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9\") pod \"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8\" (UID: \"50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8\") "
Nov 01 06:22:59.655044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.655025    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:22:59.655173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.655046    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwd82\" (UniqueName: \"kubernetes.io/projected/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9-kube-api-access-gwd82\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:22:59.656899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.656876    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9" pod "50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" (UID: "50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8"). InnerVolumeSpecName "kube-api-access-z5vn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:22:59.664067 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:22:59.663860    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:22:59.756256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.756225    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vn9\" (UniqueName: \"kubernetes.io/projected/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8-kube-api-access-z5vn9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:22:59.977053 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.976963    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7689-899601"]
Nov 01 06:22:59.982028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:22:59.982000    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7689-899601"]
Nov 01 06:23:00.366417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.366389    4468 generic.go:356] "Generic (PLEG): container finished" podID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" containerID="2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1" exitCode=137
Nov 01 06:23:00.366585 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.366431    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:23:00.366585 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.366453    4468 scope.go:122] "RemoveContainer" containerID="2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1"
Nov 01 06:23:00.367900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.367884    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"
Nov 01 06:23:00.368027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.368006    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7689-899601" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" containerName="test-curl-7689-899601" containerID="cri-o://6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b" gracePeriod=2
Nov 01 06:23:00.368109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.367899    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc" event={"ID":"cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9","Type":"ContainerDied","Data":"c703a073fd337879b33b4ad041d77553f77c16b4c5b1282c1327d3fab4fe98c6"}
Nov 01 06:23:00.368109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.368097    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c703a073fd337879b33b4ad041d77553f77c16b4c5b1282c1327d3fab4fe98c6"
Nov 01 06:23:00.376427 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.376410    4468 scope.go:122] "RemoveContainer" containerID="2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1"
Nov 01 06:23:00.376801 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:00.376722    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1\": container with ID starting with 2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1 not found: ID does not exist" containerID="2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1"
Nov 01 06:23:00.376801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.376749    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1"} err="failed to get container status \"2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1\": rpc error: code = NotFound desc = could not find container \"2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1\": container with ID starting with 2e5c928bb6f3295ee8ad44294a51472a9c41e560436d52fd29da15d3b4746ba1 not found: ID does not exist"
Nov 01 06:23:00.417227 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:00.417200    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:23:00.467794 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:00.467755    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-6a6333\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" pod="opendatahub/test-curl-7689-6a6333"
Nov 01 06:23:00.569958 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:00.569922    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-899601\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" pod="opendatahub/test-curl-7689-899601"
Nov 01 06:23:00.589234 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:00.589206    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8" path="/var/lib/kubelet/pods/50a7805c-f886-4e5c-aa5f-f9a64cc8c1a8/volumes"
Nov 01 06:23:02.280258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280227    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"]
Nov 01 06:23:02.280785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280770    4468 cpu_manager.go:414] "containerMap: removing container" podUID="cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" containerName="revoke-keys"
Nov 01 06:23:02.280830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280786    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" containerName="revoke-keys"
Nov 01 06:23:02.280830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280812    4468 cpu_manager.go:414] "containerMap: removing container" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" containerName="test-curl-7689-899601"
Nov 01 06:23:02.280830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280818    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" containerName="test-curl-7689-899601"
Nov 01 06:23:02.280922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280881    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" containerName="test-curl-7689-899601"
Nov 01 06:23:02.280922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.280894    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" containerName="revoke-keys"
Nov 01 06:23:02.285293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.285276    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.383782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.383752    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.383916 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.383792    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.383999    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.384083    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.384110    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.384162    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.384187    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdvf\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.384234    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.384301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.384255    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.393573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.393549    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"]
Nov 01 06:23:02.455513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.447120    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c79abe-openshift-default-dockercfg-cz9c6\""
Nov 01 06:23:02.485609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485574    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.485609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485615    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.485840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485684    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.485840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485730    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.485840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485760    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.485840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485797    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.485840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.485822    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdvf\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486086 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486011    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486086 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486057    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486086 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486062    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486383    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486570    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486628    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.486662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.486643    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.488241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.488214    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.488326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.488304    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.551414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.549619    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdvf\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.551414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.550075    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token\") pod \"e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") " pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.597230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.597210    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:02.605217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.605194    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-899601"
Nov 01 06:23:02.687983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.687954    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f\" (UniqueName: \"kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f\") pod \"db6f4565-1da1-4c79-83b9-53beab67b8a2\" (UID: \"db6f4565-1da1-4c79-83b9-53beab67b8a2\") "
Nov 01 06:23:02.690065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.690034    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f" pod "db6f4565-1da1-4c79-83b9-53beab67b8a2" (UID: "db6f4565-1da1-4c79-83b9-53beab67b8a2"). InnerVolumeSpecName "kube-api-access-njb9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:02.789226 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:02.789161    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njb9f\" (UniqueName: \"kubernetes.io/projected/db6f4565-1da1-4c79-83b9-53beab67b8a2-kube-api-access-njb9f\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.946636 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:02.946599    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:23:03.018744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.018722    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"]
Nov 01 06:23:03.020357 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:23:03.020336    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432412b2_d95a_4980_b775_494dee7a8f7c.slice/crio-43d1734ed0859ab5b26f87b8876329e95dcb32fbbfca947e5adc09de2922c6c1 WatchSource:0}: Error finding container 43d1734ed0859ab5b26f87b8876329e95dcb32fbbfca947e5adc09de2922c6c1: Status 404 returned error can't find the container with id 43d1734ed0859ab5b26f87b8876329e95dcb32fbbfca947e5adc09de2922c6c1
Nov 01 06:23:03.022696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.022662    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:03.022769 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.022726    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:03.022769 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.022754    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:03.391035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.391000    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" event={"ID":"432412b2-d95a-4980-b775-494dee7a8f7c","Type":"ContainerStarted","Data":"ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"}
Nov 01 06:23:03.391434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.391061    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" event={"ID":"432412b2-d95a-4980-b775-494dee7a8f7c","Type":"ContainerStarted","Data":"43d1734ed0859ab5b26f87b8876329e95dcb32fbbfca947e5adc09de2922c6c1"}
Nov 01 06:23:03.392322 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.392298    4468 generic.go:356] "Generic (PLEG): container finished" podID="db6f4565-1da1-4c79-83b9-53beab67b8a2" containerID="6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b" exitCode=137
Nov 01 06:23:03.392440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.392339    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7689-899601"
Nov 01 06:23:03.392440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.392359    4468 scope.go:122] "RemoveContainer" containerID="6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b"
Nov 01 06:23:03.401168 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.401149    4468 scope.go:122] "RemoveContainer" containerID="6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b"
Nov 01 06:23:03.401400 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:03.401381    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b\": container with ID starting with 6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b not found: ID does not exist" containerID="6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b"
Nov 01 06:23:03.401445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.401410    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b"} err="failed to get container status \"6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b\": rpc error: code = NotFound desc = could not find container \"6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b\": container with ID starting with 6e43357a6687ba69dad1ed7d137b869aebe3d53e7beae812a7aa3774efef3b9b not found: ID does not exist"
Nov 01 06:23:03.508773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.508718    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" podStartSLOduration=1.508704502 podStartE2EDuration="1.508704502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:03.507426613 +0000 UTC m=+795.075233502" watchObservedRunningTime="2026-11-01 06:23:03.508704502 +0000 UTC m=+795.076511330"
Nov 01 06:23:03.558222 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:03.558184    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7689-899601\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" pod="opendatahub/test-curl-7689-899601"
Nov 01 06:23:03.597494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:03.597444    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:04.589775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:04.589741    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6f4565-1da1-4c79-83b9-53beab67b8a2" path="/var/lib/kubelet/pods/db6f4565-1da1-4c79-83b9-53beab67b8a2/volumes"
Nov 01 06:23:04.603530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:04.603497    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:05.404155 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:05.404122    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:05.405166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:05.405142    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:23:07.516685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:07.516653    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:23:15.625731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.625704    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"]
Nov 01 06:23:15.633184 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.633166    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.685706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.685683    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"]
Nov 01 06:23:15.732398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.732374    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:23:15.734411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.734386    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:23:15.734411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.734398    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:23:15.734411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.734391    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:23:15.738024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.738005    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndmw\" (UniqueName: \"kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.738084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.738063    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.738202 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.738185    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.839550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.839517    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndmw\" (UniqueName: \"kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.839735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.839585    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.839735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.839631    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.840216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.840192    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.842201 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.842178    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.896230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.896169    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndmw\" (UniqueName: \"kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw\") pod \"maas-api-revoke-sub-a2e11976-qq9js\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:15.946663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:15.946637    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:16.317850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:16.317829    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"]
Nov 01 06:23:16.320220 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:23:16.320188    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bc217b_862d_428a_8bc0_3991f32a288e.slice/crio-b2eaccc43eb0d1303e73a215b60f91e77b66ba78f1af8f335bd52e00ea2a8615 WatchSource:0}: Error finding container b2eaccc43eb0d1303e73a215b60f91e77b66ba78f1af8f335bd52e00ea2a8615: Status 404 returned error can't find the container with id b2eaccc43eb0d1303e73a215b60f91e77b66ba78f1af8f335bd52e00ea2a8615
Nov 01 06:23:16.476225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:16.476192    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js" event={"ID":"11bc217b-862d-428a-8bc0-3991f32a288e","Type":"ContainerStarted","Data":"b2eaccc43eb0d1303e73a215b60f91e77b66ba78f1af8f335bd52e00ea2a8615"}
Nov 01 06:23:17.483740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:17.483646    4468 generic.go:356] "Generic (PLEG): container finished" podID="11bc217b-862d-428a-8bc0-3991f32a288e" containerID="be4f5e2a3bada792487cda30bd0bb500c802fe2ed1c2a98d1da9e8bc041aee98" exitCode=0
Nov 01 06:23:17.484147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:17.483738    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js" event={"ID":"11bc217b-862d-428a-8bc0-3991f32a288e","Type":"ContainerDied","Data":"be4f5e2a3bada792487cda30bd0bb500c802fe2ed1c2a98d1da9e8bc041aee98"}
Nov 01 06:23:18.649380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.649358    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:18.668791 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.668771    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw\" (UniqueName: \"kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw\") pod \"11bc217b-862d-428a-8bc0-3991f32a288e\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") "
Nov 01 06:23:18.668950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.668799    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca\") pod \"11bc217b-862d-428a-8bc0-3991f32a288e\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") "
Nov 01 06:23:18.668950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.668882    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca\") pod \"11bc217b-862d-428a-8bc0-3991f32a288e\" (UID: \"11bc217b-862d-428a-8bc0-3991f32a288e\") "
Nov 01 06:23:18.669216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.669194    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca" pod "11bc217b-862d-428a-8bc0-3991f32a288e" (UID: "11bc217b-862d-428a-8bc0-3991f32a288e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:18.670804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.670779    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw" pod "11bc217b-862d-428a-8bc0-3991f32a288e" (UID: "11bc217b-862d-428a-8bc0-3991f32a288e"). InnerVolumeSpecName "kube-api-access-lndmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:18.670891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.670858    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca" pod "11bc217b-862d-428a-8bc0-3991f32a288e" (UID: "11bc217b-862d-428a-8bc0-3991f32a288e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:23:18.769945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.769888    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lndmw\" (UniqueName: \"kubernetes.io/projected/11bc217b-862d-428a-8bc0-3991f32a288e-kube-api-access-lndmw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:18.769945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.769913    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:18.769945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:18.769922    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc217b-862d-428a-8bc0-3991f32a288e-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:19.495182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:19.495151    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"
Nov 01 06:23:19.495371 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:19.495190    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js" event={"ID":"11bc217b-862d-428a-8bc0-3991f32a288e","Type":"ContainerDied","Data":"b2eaccc43eb0d1303e73a215b60f91e77b66ba78f1af8f335bd52e00ea2a8615"}
Nov 01 06:23:19.495371 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:19.495262    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2eaccc43eb0d1303e73a215b60f91e77b66ba78f1af8f335bd52e00ea2a8615"
Nov 01 06:23:25.921206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.921170    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"]
Nov 01 06:23:25.921873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.921853    4468 cpu_manager.go:414] "containerMap: removing container" podUID="11bc217b-862d-428a-8bc0-3991f32a288e" containerName="revoke-keys"
Nov 01 06:23:25.921954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.921875    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11bc217b-862d-428a-8bc0-3991f32a288e" containerName="revoke-keys"
Nov 01 06:23:25.922025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.922005    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="11bc217b-862d-428a-8bc0-3991f32a288e" containerName="revoke-keys"
Nov 01 06:23:25.925499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.925459    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944187    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944226    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944310    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944366    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944428    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944518    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6rt\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944547    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944562    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.944626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.944579    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:25.977204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:25.977178    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ba2f3-openshift-default-dockercfg-nrd89\""
Nov 01 06:23:26.029447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.029421    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"]
Nov 01 06:23:26.045681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045644    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6rt\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.045824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045688    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.045824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045711    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.045946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045827    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.045946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045897    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.045946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045939    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.045992    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046080    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046109    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046347 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046324    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046372    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046371    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046532    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.046631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.046612    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.048310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.048281    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.049098 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.049072    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.101625 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.101601    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.101727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.101707    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6rt\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.238929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.238851    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:26.666573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.666543    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"]
Nov 01 06:23:26.668041 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:23:26.668011    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb219425_e174_40b6_b762_678a4eb173b1.slice/crio-73f85db794cbff5349bf39c559f972c357c195baebc4a8ff2a5c350c231c0d78 WatchSource:0}: Error finding container 73f85db794cbff5349bf39c559f972c357c195baebc4a8ff2a5c350c231c0d78: Status 404 returned error can't find the container with id 73f85db794cbff5349bf39c559f972c357c195baebc4a8ff2a5c350c231c0d78
Nov 01 06:23:26.670243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.670212    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:26.670302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.670279    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:26.670337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:26.670309    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:27.542554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:27.542518    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" event={"ID":"fb219425-e174-40b6-b762-678a4eb173b1","Type":"ContainerStarted","Data":"1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290"}
Nov 01 06:23:27.543066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:27.542573    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" event={"ID":"fb219425-e174-40b6-b762-678a4eb173b1","Type":"ContainerStarted","Data":"73f85db794cbff5349bf39c559f972c357c195baebc4a8ff2a5c350c231c0d78"}
Nov 01 06:23:27.729840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:27.729775    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podStartSLOduration=2.729755788 podStartE2EDuration="2.729755788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:27.725065036 +0000 UTC m=+819.292871867" watchObservedRunningTime="2026-11-01 06:23:27.729755788 +0000 UTC m=+819.297562617"
Nov 01 06:23:28.239632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:28.239601    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:28.240956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:28.240933    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 06:23:28.241009 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:28.240980    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 06:23:29.239558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:29.239518    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 06:23:29.239971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:29.239591    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 06:23:30.240101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:30.240064    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 06:23:30.240554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:30.240118    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 06:23:31.239927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:31.239891    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 06:23:31.240116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:31.239945    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 06:23:32.239546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:32.239517    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 06:23:32.239931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:32.239568    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 06:23:32.948245 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:23:32.948202    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:23:33.243700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.243620    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:33.244143 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.243983    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:33.244559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.244541    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:23:33.570508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.570455    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"]
Nov 01 06:23:33.574252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.574229    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.628033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.628012    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:23:33.628296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.628267    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:23:33.628868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.628849    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:23:33.628978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.628958    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:23:33.630907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.630882    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"]
Nov 01 06:23:33.646680 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.646648    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.646925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.646894    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.647081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.647044    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvsf\" (UniqueName: \"kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.748655 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.748626    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvsf\" (UniqueName: \"kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.748821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.748763    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.748821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.748804    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.749447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.749424    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.751188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.751163    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.806042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.806017    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvsf\" (UniqueName: \"kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf\") pod \"maas-api-revoke-sub-71e1de3d-bww8z\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:33.889687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:33.889613    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:34.261531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:34.261508    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"]
Nov 01 06:23:34.263370 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:23:34.263344    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dd073e_2167_4589_ba0c_3249614fe832.slice/crio-da8ebd16a1d692fc8f85a4429f925519f7897349e2a996a7828b0492347f62e0 WatchSource:0}: Error finding container da8ebd16a1d692fc8f85a4429f925519f7897349e2a996a7828b0492347f62e0: Status 404 returned error can't find the container with id da8ebd16a1d692fc8f85a4429f925519f7897349e2a996a7828b0492347f62e0
Nov 01 06:23:34.589247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:34.589205    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z" event={"ID":"f2dd073e-2167-4589-ba0c-3249614fe832","Type":"ContainerStarted","Data":"ba0a9fe1e67ef234398406389b18b4adf699ce5078581a1dc890d2522929c53b"}
Nov 01 06:23:34.589410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:34.589260    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z" event={"ID":"f2dd073e-2167-4589-ba0c-3249614fe832","Type":"ContainerStarted","Data":"da8ebd16a1d692fc8f85a4429f925519f7897349e2a996a7828b0492347f62e0"}
Nov 01 06:23:34.697179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:34.697123    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z" podStartSLOduration=1.697107046 podStartE2EDuration="1.697107046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:34.697014031 +0000 UTC m=+826.264820864" watchObservedRunningTime="2026-11-01 06:23:34.697107046 +0000 UTC m=+826.264913875"
Nov 01 06:23:35.590063 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:35.590025    4468 generic.go:356] "Generic (PLEG): container finished" podID="f2dd073e-2167-4589-ba0c-3249614fe832" containerID="ba0a9fe1e67ef234398406389b18b4adf699ce5078581a1dc890d2522929c53b" exitCode=0
Nov 01 06:23:35.590500 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:35.590111    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z" event={"ID":"f2dd073e-2167-4589-ba0c-3249614fe832","Type":"ContainerDied","Data":"ba0a9fe1e67ef234398406389b18b4adf699ce5078581a1dc890d2522929c53b"}
Nov 01 06:23:36.732107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.732082    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:36.885385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.885299    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf\" (UniqueName: \"kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf\") pod \"f2dd073e-2167-4589-ba0c-3249614fe832\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") "
Nov 01 06:23:36.885589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.885384    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca\") pod \"f2dd073e-2167-4589-ba0c-3249614fe832\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") "
Nov 01 06:23:36.885589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.885523    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca\") pod \"f2dd073e-2167-4589-ba0c-3249614fe832\" (UID: \"f2dd073e-2167-4589-ba0c-3249614fe832\") "
Nov 01 06:23:36.885793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.885762    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca" pod "f2dd073e-2167-4589-ba0c-3249614fe832" (UID: "f2dd073e-2167-4589-ba0c-3249614fe832"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:36.885998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.885976    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:36.887636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.887613    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca" pod "f2dd073e-2167-4589-ba0c-3249614fe832" (UID: "f2dd073e-2167-4589-ba0c-3249614fe832"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:23:36.887636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.887621    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf" pod "f2dd073e-2167-4589-ba0c-3249614fe832" (UID: "f2dd073e-2167-4589-ba0c-3249614fe832"). InnerVolumeSpecName "kube-api-access-vcvsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:36.987584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.987557    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcvsf\" (UniqueName: \"kubernetes.io/projected/f2dd073e-2167-4589-ba0c-3249614fe832-kube-api-access-vcvsf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:36.987584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:36.987581    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dd073e-2167-4589-ba0c-3249614fe832-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:37.601421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:37.601393    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"
Nov 01 06:23:37.601617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:37.601383    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z" event={"ID":"f2dd073e-2167-4589-ba0c-3249614fe832","Type":"ContainerDied","Data":"da8ebd16a1d692fc8f85a4429f925519f7897349e2a996a7828b0492347f62e0"}
Nov 01 06:23:37.601617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:37.601516    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da8ebd16a1d692fc8f85a4429f925519f7897349e2a996a7828b0492347f62e0"
Nov 01 06:23:38.500015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:38.499984    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:23:46.529573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.529544    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"]
Nov 01 06:23:46.530001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.529989    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f2dd073e-2167-4589-ba0c-3249614fe832" containerName="revoke-keys"
Nov 01 06:23:46.530043 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.530002    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dd073e-2167-4589-ba0c-3249614fe832" containerName="revoke-keys"
Nov 01 06:23:46.530110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.530101    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dd073e-2167-4589-ba0c-3249614fe832" containerName="revoke-keys"
Nov 01 06:23:46.533497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.533454    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.587738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.587312    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef9b55\"/\"openshift-service-ca.crt\""
Nov 01 06:23:46.592118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.592091    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.592260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.592131    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.592260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.592166    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.592260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.592188    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mtm\" (UniqueName: \"kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.592499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.592453    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.592586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.592573    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.636779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.636750    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef9b55\"/\"kube-root-ca.crt\""
Nov 01 06:23:46.636974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.636885    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef9b55\"/\"default-dockercfg-8459r\""
Nov 01 06:23:46.636974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.636899    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef9b55\"/\"test-model-ef9b55-kserve-self-signed-certs\""
Nov 01 06:23:46.639394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.639379    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"]
Nov 01 06:23:46.695071 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.695037    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.695281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.695083    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.695281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.695119    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.695281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.695139    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mtm\" (UniqueName: \"kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.695281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.695211    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.695281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.695275    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.697014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.696487    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.697014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.696751    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.697014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.696978    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.700884 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.700859    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.701926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.701901    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.753198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.753169    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mtm\" (UniqueName: \"kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm\") pod \"test-model-ef9b55-kserve-77fb678b7-pjrv8\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") " pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:46.845523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:46.845491    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:47.217685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:47.217661    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"]
Nov 01 06:23:47.218752 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:23:47.218728    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod159db47b_55fb_423a_821b_7567be5cd07d.slice/crio-803c63e294ba7e26e4793137908c53ef5596c67aaa018bbb1492aff34d058b62 WatchSource:0}: Error finding container 803c63e294ba7e26e4793137908c53ef5596c67aaa018bbb1492aff34d058b62: Status 404 returned error can't find the container with id 803c63e294ba7e26e4793137908c53ef5596c67aaa018bbb1492aff34d058b62
Nov 01 06:23:47.655159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:47.655126    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" event={"ID":"159db47b-55fb-423a-821b-7567be5cd07d","Type":"ContainerStarted","Data":"e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c"}
Nov 01 06:23:47.655159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:47.655172    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" event={"ID":"159db47b-55fb-423a-821b-7567be5cd07d","Type":"ContainerStarted","Data":"803c63e294ba7e26e4793137908c53ef5596c67aaa018bbb1492aff34d058b62"}
Nov 01 06:23:47.770228 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:47.770176    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" podStartSLOduration=1.770158117 podStartE2EDuration="1.770158117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:47.768421919 +0000 UTC m=+839.336228750" watchObservedRunningTime="2026-11-01 06:23:47.770158117 +0000 UTC m=+839.337965000"
Nov 01 06:23:47.845786 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:47.845756    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:56.846287 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:56.846248    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:56.859109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:56.859084    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:57.720149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:57.720122    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:23:58.716292 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:58.716194    4468 generic.go:356] "Generic (PLEG): container finished" podID="432412b2-d95a-4980-b775-494dee7a8f7c" containerID="ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64" exitCode=0
Nov 01 06:23:58.716292 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:58.716275    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" event={"ID":"432412b2-d95a-4980-b775-494dee7a8f7c","Type":"ContainerDied","Data":"ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"}
Nov 01 06:23:58.716752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:58.716704    4468 scope.go:122] "RemoveContainer" containerID="ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"
Nov 01 06:23:58.717913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:58.717882    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:58.718029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:58.717961    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:58.718029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:58.718006    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:23:59.733352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:23:59.733269    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" event={"ID":"432412b2-d95a-4980-b775-494dee7a8f7c","Type":"ContainerStarted","Data":"0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34"}
Nov 01 06:24:00.598375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:00.598300    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:24:00.599498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:00.599458    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 06:24:00.599634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:00.599516    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 06:24:01.598579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:01.598550    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 06:24:01.598967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:01.598597    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 06:24:02.037871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:02.037786    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:24:02.598124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:02.598095    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 06:24:02.598281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:02.598145    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 06:24:02.966637 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:24:02.966548    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:24:03.598523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:03.598459    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 06:24:03.598729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:03.598548    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 06:24:04.628312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:04.628280    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:24:04.628862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:04.628752    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:24:04.629736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:04.629717    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 06:24:12.674942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.674908    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"]
Nov 01 06:24:12.677646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.677631    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.731849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.731820    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:24:12.732171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.732083    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:24:12.732569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.732542    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:24:12.781570 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.781543    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:24:12.784142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.784123    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"]
Nov 01 06:24:12.786093 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.786071    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.786208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.786189    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.786277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.786267    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmz2\" (UniqueName: \"kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.886935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.886908    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmz2\" (UniqueName: \"kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.887111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.886984    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.887111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.887040    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.887623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.887605    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.889413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.889389    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.943333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.943267    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmz2\" (UniqueName: \"kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2\") pod \"maas-api-revoke-sub-f303bf6a-25cpr\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:12.989445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:12.989415    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:13.358667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:13.358640    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"]
Nov 01 06:24:13.360262 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:13.360236    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976fcff3_f87a_40f0_97ce_acbc87b90a3d.slice/crio-3336126c39aa5cceaad0df5b8a5afffb781ef21b02d08c2802d5746921fa7765 WatchSource:0}: Error finding container 3336126c39aa5cceaad0df5b8a5afffb781ef21b02d08c2802d5746921fa7765: Status 404 returned error can't find the container with id 3336126c39aa5cceaad0df5b8a5afffb781ef21b02d08c2802d5746921fa7765
Nov 01 06:24:13.846011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:13.845976    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr" event={"ID":"976fcff3-f87a-40f0-97ce-acbc87b90a3d","Type":"ContainerStarted","Data":"71914a5909c51f41f05a642d09efd89bc4a1ab779d14803dc0a6fb07f8b1a6d7"}
Nov 01 06:24:13.846420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:13.846031    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr" event={"ID":"976fcff3-f87a-40f0-97ce-acbc87b90a3d","Type":"ContainerStarted","Data":"3336126c39aa5cceaad0df5b8a5afffb781ef21b02d08c2802d5746921fa7765"}
Nov 01 06:24:13.958702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:13.958593    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr" podStartSLOduration=1.958577107 podStartE2EDuration="1.958577107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:13.957410573 +0000 UTC m=+865.525217440" watchObservedRunningTime="2026-11-01 06:24:13.958577107 +0000 UTC m=+865.526383936"
Nov 01 06:24:14.853173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:14.853134    4468 generic.go:356] "Generic (PLEG): container finished" podID="976fcff3-f87a-40f0-97ce-acbc87b90a3d" containerID="71914a5909c51f41f05a642d09efd89bc4a1ab779d14803dc0a6fb07f8b1a6d7" exitCode=0
Nov 01 06:24:14.853589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:14.853189    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr" event={"ID":"976fcff3-f87a-40f0-97ce-acbc87b90a3d","Type":"ContainerDied","Data":"71914a5909c51f41f05a642d09efd89bc4a1ab779d14803dc0a6fb07f8b1a6d7"}
Nov 01 06:24:16.002140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.002115    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:16.123885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.123808    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca\") pod \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") "
Nov 01 06:24:16.123885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.123864    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2\" (UniqueName: \"kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2\") pod \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") "
Nov 01 06:24:16.124109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.123893    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca\") pod \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\" (UID: \"976fcff3-f87a-40f0-97ce-acbc87b90a3d\") "
Nov 01 06:24:16.124328 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.124191    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca" pod "976fcff3-f87a-40f0-97ce-acbc87b90a3d" (UID: "976fcff3-f87a-40f0-97ce-acbc87b90a3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:24:16.124328 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.124303    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:16.127144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.127117    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca" pod "976fcff3-f87a-40f0-97ce-acbc87b90a3d" (UID: "976fcff3-f87a-40f0-97ce-acbc87b90a3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:24:16.127250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.127147    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2" pod "976fcff3-f87a-40f0-97ce-acbc87b90a3d" (UID: "976fcff3-f87a-40f0-97ce-acbc87b90a3d"). InnerVolumeSpecName "kube-api-access-jnmz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:24:16.224956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.224928    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmz2\" (UniqueName: \"kubernetes.io/projected/976fcff3-f87a-40f0-97ce-acbc87b90a3d-kube-api-access-jnmz2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:16.224956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.224952    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976fcff3-f87a-40f0-97ce-acbc87b90a3d-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:16.866530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.866445    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"
Nov 01 06:24:16.866677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.866446    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr" event={"ID":"976fcff3-f87a-40f0-97ce-acbc87b90a3d","Type":"ContainerDied","Data":"3336126c39aa5cceaad0df5b8a5afffb781ef21b02d08c2802d5746921fa7765"}
Nov 01 06:24:16.866677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:16.866608    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3336126c39aa5cceaad0df5b8a5afffb781ef21b02d08c2802d5746921fa7765"
Nov 01 06:24:19.133808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.133773    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"]
Nov 01 06:24:19.134348 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.134284    4468 cpu_manager.go:414] "containerMap: removing container" podUID="976fcff3-f87a-40f0-97ce-acbc87b90a3d" containerName="revoke-keys"
Nov 01 06:24:19.134348 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.134300    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976fcff3-f87a-40f0-97ce-acbc87b90a3d" containerName="revoke-keys"
Nov 01 06:24:19.134417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.134408    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="976fcff3-f87a-40f0-97ce-acbc87b90a3d" containerName="revoke-keys"
Nov 01 06:24:19.136738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.136722    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.246264 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.246229    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"]
Nov 01 06:24:19.266281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266247    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266364    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266528    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266585 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266561    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266608    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266712 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266689    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266758    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266816    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kg4\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.266892 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.266856    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.291899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.291866    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-dc6e48-openshift-default-dockercfg-6n7lp\""
Nov 01 06:24:19.368376 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368343    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368392    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368439    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368458    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368519    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368544    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368578    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368601    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kg4\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.368829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368820    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.369013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368818    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.369013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368889    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.369013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.368919    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.369170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.369106    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.369353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.369334    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.371025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.371002    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.371104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.371049    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.425051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.424989    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kg4\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.425165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.425074    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.448900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.448860    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:19.872490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.872447    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"]
Nov 01 06:24:19.873445 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:19.873412    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646fd28f_2ae4_48ac_931f_b3f12abad15e.slice/crio-cc8df0020c98f0e3615c9be70944198b6795b45054a45bb3810489077b9d4010 WatchSource:0}: Error finding container cc8df0020c98f0e3615c9be70944198b6795b45054a45bb3810489077b9d4010: Status 404 returned error can't find the container with id cc8df0020c98f0e3615c9be70944198b6795b45054a45bb3810489077b9d4010
Nov 01 06:24:19.875597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.875565    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:24:19.875690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.875627    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:24:19.875690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.875658    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:24:19.883417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:19.883389    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" event={"ID":"646fd28f-2ae4-48ac-931f-b3f12abad15e","Type":"ContainerStarted","Data":"cc8df0020c98f0e3615c9be70944198b6795b45054a45bb3810489077b9d4010"}
Nov 01 06:24:20.890926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:20.890894    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" event={"ID":"646fd28f-2ae4-48ac-931f-b3f12abad15e","Type":"ContainerStarted","Data":"d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"}
Nov 01 06:24:21.057586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:21.057538    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podStartSLOduration=2.057524709 podStartE2EDuration="2.057524709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:21.006964432 +0000 UTC m=+872.574771289" watchObservedRunningTime="2026-11-01 06:24:21.057524709 +0000 UTC m=+872.625331537"
Nov 01 06:24:21.449767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:21.449738    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:21.455405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:21.455382    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:21.899910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:21.899878    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:21.900748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:21.900728    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:24:32.946385 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:24:32.946342    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:24:36.951831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:36.951800    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"]
Nov 01 06:24:36.955498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:36.955455    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.058893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.058845    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:24:37.058893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.058878    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:24:37.059099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.058897    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:24:37.059099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.058908    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:24:37.061672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.061645    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"]
Nov 01 06:24:37.087851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.087824    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.087967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.087900    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65ng\" (UniqueName: \"kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.088066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.088054    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.189355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.189317    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65ng\" (UniqueName: \"kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.189555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.189449    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.189555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.189536    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.190037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.190019    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.192007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.191985    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.245521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.245436    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65ng\" (UniqueName: \"kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng\") pod \"maas-api-revoke-sub-1d53aedf-2zmtg\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.268306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.268278    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:37.592786 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.592756    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"]
Nov 01 06:24:37.596790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.596773    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.643669 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.643641    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"]
Nov 01 06:24:37.647357 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:37.647313    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a916d09_6233_4ebc_aaa4_c159222fea79.slice/crio-c87a418c1734b7b4e1d71dae11eb3e6fb204e13a4005e3df211ded80bc6fed24 WatchSource:0}: Error finding container c87a418c1734b7b4e1d71dae11eb3e6fb204e13a4005e3df211ded80bc6fed24: Status 404 returned error can't find the container with id c87a418c1734b7b4e1d71dae11eb3e6fb204e13a4005e3df211ded80bc6fed24
Nov 01 06:24:37.650202 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.650159    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"kube-root-ca.crt\""
Nov 01 06:24:37.650341 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.650296    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"distinct-2-w4-190724-kserve-self-signed-certs\""
Nov 01 06:24:37.651637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.651616    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"openshift-service-ca.crt\""
Nov 01 06:24:37.651804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.650600    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"default-dockercfg-52ts2\""
Nov 01 06:24:37.654605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.654582    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"]
Nov 01 06:24:37.694673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.694648    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.694798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.694693    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.694798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.694783    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.694887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.694843    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.694945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.694932    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.694982 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.694954    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn77\" (UniqueName: \"kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796291 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796252    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796317    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796378    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796448    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796505    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn77\" (UniqueName: \"kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796607    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796748    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796834 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796795    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.796986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.796969    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.798768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.798742    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.799025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.799009    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.853638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.853575    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn77\" (UniqueName: \"kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77\") pod \"distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") " pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:37.908628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:37.908601    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:38.017892 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:38.017857    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg" event={"ID":"2a916d09-6233-4ebc-aaa4-c159222fea79","Type":"ContainerStarted","Data":"dd6c239e792849b099df18f41c9abfab7cabf9b743864911fbb09f42165d0bb2"}
Nov 01 06:24:38.018277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:38.017909    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg" event={"ID":"2a916d09-6233-4ebc-aaa4-c159222fea79","Type":"ContainerStarted","Data":"c87a418c1734b7b4e1d71dae11eb3e6fb204e13a4005e3df211ded80bc6fed24"}
Nov 01 06:24:38.130440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:38.130346    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg" podStartSLOduration=2.130332209 podStartE2EDuration="2.130332209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:38.12976594 +0000 UTC m=+889.697572769" watchObservedRunningTime="2026-11-01 06:24:38.130332209 +0000 UTC m=+889.698139037"
Nov 01 06:24:38.280164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:38.280138    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"]
Nov 01 06:24:38.281725 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:38.281695    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30617666_577d_44e0_9961_1e4978434e76.slice/crio-969bc4302219a4cf5d7ac3fc5e35a9a3dc37fdf5835b9e03a124e2d8c06bfdc9 WatchSource:0}: Error finding container 969bc4302219a4cf5d7ac3fc5e35a9a3dc37fdf5835b9e03a124e2d8c06bfdc9: Status 404 returned error can't find the container with id 969bc4302219a4cf5d7ac3fc5e35a9a3dc37fdf5835b9e03a124e2d8c06bfdc9
Nov 01 06:24:39.025207 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:39.025169    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" event={"ID":"30617666-577d-44e0-9961-1e4978434e76","Type":"ContainerStarted","Data":"411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95"}
Nov 01 06:24:39.025674 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:39.025213    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" event={"ID":"30617666-577d-44e0-9961-1e4978434e76","Type":"ContainerStarted","Data":"969bc4302219a4cf5d7ac3fc5e35a9a3dc37fdf5835b9e03a124e2d8c06bfdc9"}
Nov 01 06:24:39.026868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:39.026844    4468 generic.go:356] "Generic (PLEG): container finished" podID="2a916d09-6233-4ebc-aaa4-c159222fea79" containerID="dd6c239e792849b099df18f41c9abfab7cabf9b743864911fbb09f42165d0bb2" exitCode=0
Nov 01 06:24:39.026991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:39.026877    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg" event={"ID":"2a916d09-6233-4ebc-aaa4-c159222fea79","Type":"ContainerDied","Data":"dd6c239e792849b099df18f41c9abfab7cabf9b743864911fbb09f42165d0bb2"}
Nov 01 06:24:39.141651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:39.140193    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podStartSLOduration=2.140176554 podStartE2EDuration="2.140176554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:39.139871847 +0000 UTC m=+890.707678678" watchObservedRunningTime="2026-11-01 06:24:39.140176554 +0000 UTC m=+890.707983385"
Nov 01 06:24:39.909249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:39.909216    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:40.185851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.185824    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:40.338205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.338176    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca\") pod \"2a916d09-6233-4ebc-aaa4-c159222fea79\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") "
Nov 01 06:24:40.338395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.338221    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca\") pod \"2a916d09-6233-4ebc-aaa4-c159222fea79\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") "
Nov 01 06:24:40.338395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.338290    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng\" (UniqueName: \"kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng\") pod \"2a916d09-6233-4ebc-aaa4-c159222fea79\" (UID: \"2a916d09-6233-4ebc-aaa4-c159222fea79\") "
Nov 01 06:24:40.338698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.338667    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca" pod "2a916d09-6233-4ebc-aaa4-c159222fea79" (UID: "2a916d09-6233-4ebc-aaa4-c159222fea79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:24:40.338908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.338879    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:40.340453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.340431    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca" pod "2a916d09-6233-4ebc-aaa4-c159222fea79" (UID: "2a916d09-6233-4ebc-aaa4-c159222fea79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:24:40.340958 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.340936    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng" pod "2a916d09-6233-4ebc-aaa4-c159222fea79" (UID: "2a916d09-6233-4ebc-aaa4-c159222fea79"). InnerVolumeSpecName "kube-api-access-t65ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:24:40.439813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.439734    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t65ng\" (UniqueName: \"kubernetes.io/projected/2a916d09-6233-4ebc-aaa4-c159222fea79-kube-api-access-t65ng\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:40.439813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.439763    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a916d09-6233-4ebc-aaa4-c159222fea79-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:40.601115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.601087    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"]
Nov 01 06:24:40.601637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.601620    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2a916d09-6233-4ebc-aaa4-c159222fea79" containerName="revoke-keys"
Nov 01 06:24:40.601730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.601638    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a916d09-6233-4ebc-aaa4-c159222fea79" containerName="revoke-keys"
Nov 01 06:24:40.601804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.601790    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a916d09-6233-4ebc-aaa4-c159222fea79" containerName="revoke-keys"
Nov 01 06:24:40.605025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.605008    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.661523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.661489    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"]
Nov 01 06:24:40.742783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.742696    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.742943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.742807    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.742943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.742857    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkpd\" (UniqueName: \"kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.843800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.843763    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.843995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.843831    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.843995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.843859    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkpd\" (UniqueName: \"kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.844555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.844530    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.846544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.846519    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.899856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.899824    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkpd\" (UniqueName: \"kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd\") pod \"maas-api-revoke-sub-71d0b8fa-5gzrv\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:40.916842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:40.916814    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:41.040971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:41.040875    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg" event={"ID":"2a916d09-6233-4ebc-aaa4-c159222fea79","Type":"ContainerDied","Data":"c87a418c1734b7b4e1d71dae11eb3e6fb204e13a4005e3df211ded80bc6fed24"}
Nov 01 06:24:41.040971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:41.040906    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"
Nov 01 06:24:41.040971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:41.040938    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87a418c1734b7b4e1d71dae11eb3e6fb204e13a4005e3df211ded80bc6fed24"
Nov 01 06:24:41.285626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:41.285596    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"]
Nov 01 06:24:41.286839 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:41.286813    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded882d7f_00d9_42f4_beee_e57bf1f090d9.slice/crio-d878398b5941cd348d03522dbc351f5248262fb00f996e8205975ca4c0897ae0 WatchSource:0}: Error finding container d878398b5941cd348d03522dbc351f5248262fb00f996e8205975ca4c0897ae0: Status 404 returned error can't find the container with id d878398b5941cd348d03522dbc351f5248262fb00f996e8205975ca4c0897ae0
Nov 01 06:24:42.055901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:42.055863    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv" event={"ID":"ed882d7f-00d9-42f4-beee-e57bf1f090d9","Type":"ContainerStarted","Data":"550220841f074168575ab7bf0ab606829e88da232ba20995aaeaa00368aac9c1"}
Nov 01 06:24:42.056054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:42.055944    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv" event={"ID":"ed882d7f-00d9-42f4-beee-e57bf1f090d9","Type":"ContainerStarted","Data":"d878398b5941cd348d03522dbc351f5248262fb00f996e8205975ca4c0897ae0"}
Nov 01 06:24:42.217390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:42.217338    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv" podStartSLOduration=2.217325249 podStartE2EDuration="2.217325249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:42.166924119 +0000 UTC m=+893.734730953" watchObservedRunningTime="2026-11-01 06:24:42.217325249 +0000 UTC m=+893.785132077"
Nov 01 06:24:43.064294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:43.064263    4468 generic.go:356] "Generic (PLEG): container finished" podID="ed882d7f-00d9-42f4-beee-e57bf1f090d9" containerID="550220841f074168575ab7bf0ab606829e88da232ba20995aaeaa00368aac9c1" exitCode=0
Nov 01 06:24:43.064726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:43.064350    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv" event={"ID":"ed882d7f-00d9-42f4-beee-e57bf1f090d9","Type":"ContainerDied","Data":"550220841f074168575ab7bf0ab606829e88da232ba20995aaeaa00368aac9c1"}
Nov 01 06:24:44.213987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.213961    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:44.385727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.385629    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca\") pod \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") "
Nov 01 06:24:44.385727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.385695    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd\" (UniqueName: \"kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd\") pod \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") "
Nov 01 06:24:44.385968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.385772    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca\") pod \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\" (UID: \"ed882d7f-00d9-42f4-beee-e57bf1f090d9\") "
Nov 01 06:24:44.386033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.386010    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca" pod "ed882d7f-00d9-42f4-beee-e57bf1f090d9" (UID: "ed882d7f-00d9-42f4-beee-e57bf1f090d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:24:44.386274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.386257    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:44.387728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.387710    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd" pod "ed882d7f-00d9-42f4-beee-e57bf1f090d9" (UID: "ed882d7f-00d9-42f4-beee-e57bf1f090d9"). InnerVolumeSpecName "kube-api-access-xkkpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:24:44.387852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.387839    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca" pod "ed882d7f-00d9-42f4-beee-e57bf1f090d9" (UID: "ed882d7f-00d9-42f4-beee-e57bf1f090d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:24:44.487285 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.487247    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkkpd\" (UniqueName: \"kubernetes.io/projected/ed882d7f-00d9-42f4-beee-e57bf1f090d9-kube-api-access-xkkpd\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:44.487285 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:44.487278    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed882d7f-00d9-42f4-beee-e57bf1f090d9-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:45.076831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:45.076796    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv" event={"ID":"ed882d7f-00d9-42f4-beee-e57bf1f090d9","Type":"ContainerDied","Data":"d878398b5941cd348d03522dbc351f5248262fb00f996e8205975ca4c0897ae0"}
Nov 01 06:24:45.076831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:45.076822    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"
Nov 01 06:24:45.077097 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:45.076853    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d878398b5941cd348d03522dbc351f5248262fb00f996e8205975ca4c0897ae0"
Nov 01 06:24:47.546094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:47.545342    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:24:47.909515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:47.909426    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:47.921806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:47.921779    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:48.110930 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:48.110900    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:24:52.694544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.694510    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"]
Nov 01 06:24:52.695000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.694986    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ed882d7f-00d9-42f4-beee-e57bf1f090d9" containerName="revoke-keys"
Nov 01 06:24:52.695041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.695001    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed882d7f-00d9-42f4-beee-e57bf1f090d9" containerName="revoke-keys"
Nov 01 06:24:52.695100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.695091    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed882d7f-00d9-42f4-beee-e57bf1f090d9" containerName="revoke-keys"
Nov 01 06:24:52.699602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.699587    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.756337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.756305    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"]
Nov 01 06:24:52.779692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.779635    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.779885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.779727    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48tw\" (UniqueName: \"kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.779885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.779833    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.802032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.801988    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:24:52.802032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.801992    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:24:52.802278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.801988    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:24:52.802278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.802060    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:24:52.880434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.880399    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.880668 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.880505    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48tw\" (UniqueName: \"kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.880668 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.880594    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.881194 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.881173    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.882955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.882933    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:52.936868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:52.936833    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48tw\" (UniqueName: \"kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw\") pod \"maas-api-revoke-sub-504b94f6-tmm8x\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:53.012867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:53.012781    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:53.386971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:53.386944    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"]
Nov 01 06:24:53.388061 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:53.388032    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b5c367_c1c3_4cc6_8b39_27a8443e98fd.slice/crio-fb616ed0847eaa8d9574e2d05da465e4e2110f782f20ca6f545fe39b043eed52 WatchSource:0}: Error finding container fb616ed0847eaa8d9574e2d05da465e4e2110f782f20ca6f545fe39b043eed52: Status 404 returned error can't find the container with id fb616ed0847eaa8d9574e2d05da465e4e2110f782f20ca6f545fe39b043eed52
Nov 01 06:24:54.130861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:54.130825    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x" event={"ID":"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd","Type":"ContainerStarted","Data":"d9b80fd818b55f6de4c86af56cfa37b33a50eb2dfe2e6ebf71b9298d95a7490c"}
Nov 01 06:24:54.131352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:54.130887    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x" event={"ID":"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd","Type":"ContainerStarted","Data":"fb616ed0847eaa8d9574e2d05da465e4e2110f782f20ca6f545fe39b043eed52"}
Nov 01 06:24:54.243877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:54.243817    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x" podStartSLOduration=2.243804642 podStartE2EDuration="2.243804642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:54.241867529 +0000 UTC m=+905.809674373" watchObservedRunningTime="2026-11-01 06:24:54.243804642 +0000 UTC m=+905.811611470"
Nov 01 06:24:55.138026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:55.137994    4468 generic.go:356] "Generic (PLEG): container finished" podID="d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" containerID="d9b80fd818b55f6de4c86af56cfa37b33a50eb2dfe2e6ebf71b9298d95a7490c" exitCode=0
Nov 01 06:24:55.138552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:55.138077    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x" event={"ID":"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd","Type":"ContainerDied","Data":"d9b80fd818b55f6de4c86af56cfa37b33a50eb2dfe2e6ebf71b9298d95a7490c"}
Nov 01 06:24:56.288377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.288345    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:56.318289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.318259    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca\") pod \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") "
Nov 01 06:24:56.318430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.318370    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca\") pod \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") "
Nov 01 06:24:56.318430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.318395    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw\" (UniqueName: \"kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw\") pod \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\" (UID: \"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd\") "
Nov 01 06:24:56.318806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.318778    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca" pod "d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" (UID: "d9b5c367-c1c3-4cc6-8b39-27a8443e98fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:24:56.319041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.318947    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:56.320397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.320371    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca" pod "d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" (UID: "d9b5c367-c1c3-4cc6-8b39-27a8443e98fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:24:56.320638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.320620    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw" pod "d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" (UID: "d9b5c367-c1c3-4cc6-8b39-27a8443e98fd"). InnerVolumeSpecName "kube-api-access-l48tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:24:56.419711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.419621    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:56.419711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.419648    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l48tw\" (UniqueName: \"kubernetes.io/projected/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd-kube-api-access-l48tw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:24:56.715124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.715037    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"]
Nov 01 06:24:56.715575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.715561    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" containerName="revoke-keys"
Nov 01 06:24:56.715636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.715576    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" containerName="revoke-keys"
Nov 01 06:24:56.715734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.715724    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" containerName="revoke-keys"
Nov 01 06:24:56.719182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.719167    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.776238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.776201    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"]
Nov 01 06:24:56.823541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.823511    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.823700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.823587    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz9m\" (UniqueName: \"kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.823700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.823675    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.925007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.924974    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz9m\" (UniqueName: \"kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.925210 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.925055    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.925274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.925219    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.925788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.925762    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.927640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.927621    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:56.981607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:56.981541    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz9m\" (UniqueName: \"kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m\") pod \"maas-api-revoke-sub-19aed737-ffflq\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:57.032569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:57.032535    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:24:57.150112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:57.150080    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x" event={"ID":"d9b5c367-c1c3-4cc6-8b39-27a8443e98fd","Type":"ContainerDied","Data":"fb616ed0847eaa8d9574e2d05da465e4e2110f782f20ca6f545fe39b043eed52"}
Nov 01 06:24:57.150274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:57.150128    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb616ed0847eaa8d9574e2d05da465e4e2110f782f20ca6f545fe39b043eed52"
Nov 01 06:24:57.150274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:57.150093    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"
Nov 01 06:24:57.402821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:57.402794    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"]
Nov 01 06:24:57.403517 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:24:57.403489    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279f8367_0f05_472c_a761_1ba5f3bcdf29.slice/crio-0260d77d07f7886a62edae8c0669fe2d46aef3ebcf4407a1486b8ccd468e87e8 WatchSource:0}: Error finding container 0260d77d07f7886a62edae8c0669fe2d46aef3ebcf4407a1486b8ccd468e87e8: Status 404 returned error can't find the container with id 0260d77d07f7886a62edae8c0669fe2d46aef3ebcf4407a1486b8ccd468e87e8
Nov 01 06:24:58.157531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:58.157493    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq" event={"ID":"279f8367-0f05-472c-a761-1ba5f3bcdf29","Type":"ContainerStarted","Data":"5dc5fb0a16c55d10f112c249ad83c5b245cb09ebf235d3e16c659c1777e49f84"}
Nov 01 06:24:58.157704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:58.157547    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq" event={"ID":"279f8367-0f05-472c-a761-1ba5f3bcdf29","Type":"ContainerStarted","Data":"0260d77d07f7886a62edae8c0669fe2d46aef3ebcf4407a1486b8ccd468e87e8"}
Nov 01 06:24:58.271073 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:58.271028    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq" podStartSLOduration=2.27101163 podStartE2EDuration="2.27101163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:58.26865647 +0000 UTC m=+909.836463299" watchObservedRunningTime="2026-11-01 06:24:58.27101163 +0000 UTC m=+909.838818461"
Nov 01 06:24:59.164301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:59.164264    4468 generic.go:356] "Generic (PLEG): container finished" podID="279f8367-0f05-472c-a761-1ba5f3bcdf29" containerID="5dc5fb0a16c55d10f112c249ad83c5b245cb09ebf235d3e16c659c1777e49f84" exitCode=0
Nov 01 06:24:59.164694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:24:59.164340    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq" event={"ID":"279f8367-0f05-472c-a761-1ba5f3bcdf29","Type":"ContainerDied","Data":"5dc5fb0a16c55d10f112c249ad83c5b245cb09ebf235d3e16c659c1777e49f84"}
Nov 01 06:25:00.339618 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.339588    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:25:00.467496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.467372    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m\" (UniqueName: \"kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m\") pod \"279f8367-0f05-472c-a761-1ba5f3bcdf29\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") "
Nov 01 06:25:00.467496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.467437    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca\") pod \"279f8367-0f05-472c-a761-1ba5f3bcdf29\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") "
Nov 01 06:25:00.467733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.467640    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca\") pod \"279f8367-0f05-472c-a761-1ba5f3bcdf29\" (UID: \"279f8367-0f05-472c-a761-1ba5f3bcdf29\") "
Nov 01 06:25:00.468160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.468128    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca" pod "279f8367-0f05-472c-a761-1ba5f3bcdf29" (UID: "279f8367-0f05-472c-a761-1ba5f3bcdf29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:00.481571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.479600    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca" pod "279f8367-0f05-472c-a761-1ba5f3bcdf29" (UID: "279f8367-0f05-472c-a761-1ba5f3bcdf29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:25:00.481698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.481582    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m" pod "279f8367-0f05-472c-a761-1ba5f3bcdf29" (UID: "279f8367-0f05-472c-a761-1ba5f3bcdf29"). InnerVolumeSpecName "kube-api-access-xmz9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:00.569232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.569202    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmz9m\" (UniqueName: \"kubernetes.io/projected/279f8367-0f05-472c-a761-1ba5f3bcdf29-kube-api-access-xmz9m\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:00.569232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.569228    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:00.569232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:00.569238    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8367-0f05-472c-a761-1ba5f3bcdf29-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:01.178293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:01.178255    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq" event={"ID":"279f8367-0f05-472c-a761-1ba5f3bcdf29","Type":"ContainerDied","Data":"0260d77d07f7886a62edae8c0669fe2d46aef3ebcf4407a1486b8ccd468e87e8"}
Nov 01 06:25:01.178534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:01.178295    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"
Nov 01 06:25:01.178534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:01.178305    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0260d77d07f7886a62edae8c0669fe2d46aef3ebcf4407a1486b8ccd468e87e8"
Nov 01 06:25:02.946442 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:25:02.946409    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:25:12.606737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.606700    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"]
Nov 01 06:25:12.607255 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.607199    4468 cpu_manager.go:414] "containerMap: removing container" podUID="279f8367-0f05-472c-a761-1ba5f3bcdf29" containerName="revoke-keys"
Nov 01 06:25:12.607255 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.607209    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279f8367-0f05-472c-a761-1ba5f3bcdf29" containerName="revoke-keys"
Nov 01 06:25:12.607336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.607291    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="279f8367-0f05-472c-a761-1ba5f3bcdf29" containerName="revoke-keys"
Nov 01 06:25:12.610815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.610799    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.664559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.664525    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:25:12.671810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.671772    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"]
Nov 01 06:25:12.707959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.707923    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.708160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.708078    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phd62\" (UniqueName: \"kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.708160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.708133    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.714082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.714057    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:25:12.714246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.714091    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:25:12.714246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.714152    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:25:12.809349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.809313    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phd62\" (UniqueName: \"kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.809547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.809367    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.809547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.809458    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.810069 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.810047    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.811994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.811966    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.865928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.865851    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phd62\" (UniqueName: \"kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62\") pod \"maas-api-revoke-sub-a8a32142-lh27h\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:12.926087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:12.926011    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:13.306452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:13.306420    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"]
Nov 01 06:25:13.307131 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:13.307110    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53be671d_c05a_46ca_a810_6e08191611b8.slice/crio-df95c4bf5c9c8452e06fe0e04f70cdba6b94be29c445b1aade7bca1912251973 WatchSource:0}: Error finding container df95c4bf5c9c8452e06fe0e04f70cdba6b94be29c445b1aade7bca1912251973: Status 404 returned error can't find the container with id df95c4bf5c9c8452e06fe0e04f70cdba6b94be29c445b1aade7bca1912251973
Nov 01 06:25:14.258169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:14.258129    4468 generic.go:356] "Generic (PLEG): container finished" podID="53be671d-c05a-46ca-a810-6e08191611b8" containerID="4590fe010d17d67565103cb4240b594117970127bcf06949049562471e607891" exitCode=0
Nov 01 06:25:14.258652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:14.258201    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h" event={"ID":"53be671d-c05a-46ca-a810-6e08191611b8","Type":"ContainerDied","Data":"4590fe010d17d67565103cb4240b594117970127bcf06949049562471e607891"}
Nov 01 06:25:14.258652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:14.258272    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h" event={"ID":"53be671d-c05a-46ca-a810-6e08191611b8","Type":"ContainerStarted","Data":"df95c4bf5c9c8452e06fe0e04f70cdba6b94be29c445b1aade7bca1912251973"}
Nov 01 06:25:15.410063 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.410029    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:15.541108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.541028    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca\") pod \"53be671d-c05a-46ca-a810-6e08191611b8\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") "
Nov 01 06:25:15.541108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.541098    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca\") pod \"53be671d-c05a-46ca-a810-6e08191611b8\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") "
Nov 01 06:25:15.541332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.541236    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62\" (UniqueName: \"kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62\") pod \"53be671d-c05a-46ca-a810-6e08191611b8\" (UID: \"53be671d-c05a-46ca-a810-6e08191611b8\") "
Nov 01 06:25:15.541490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.541448    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca" pod "53be671d-c05a-46ca-a810-6e08191611b8" (UID: "53be671d-c05a-46ca-a810-6e08191611b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:15.541683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.541666    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:15.543426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.543398    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca" pod "53be671d-c05a-46ca-a810-6e08191611b8" (UID: "53be671d-c05a-46ca-a810-6e08191611b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:25:15.543656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.543636    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62" pod "53be671d-c05a-46ca-a810-6e08191611b8" (UID: "53be671d-c05a-46ca-a810-6e08191611b8"). InnerVolumeSpecName "kube-api-access-phd62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:15.642969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.642931    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53be671d-c05a-46ca-a810-6e08191611b8-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:15.642969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:15.642963    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phd62\" (UniqueName: \"kubernetes.io/projected/53be671d-c05a-46ca-a810-6e08191611b8-kube-api-access-phd62\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:16.272675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:16.272640    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h" event={"ID":"53be671d-c05a-46ca-a810-6e08191611b8","Type":"ContainerDied","Data":"df95c4bf5c9c8452e06fe0e04f70cdba6b94be29c445b1aade7bca1912251973"}
Nov 01 06:25:16.272839 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:16.272701    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df95c4bf5c9c8452e06fe0e04f70cdba6b94be29c445b1aade7bca1912251973"
Nov 01 06:25:16.272839 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:16.272717    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"
Nov 01 06:25:17.331979 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:17.331938    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:25:21.229638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.229606    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"]
Nov 01 06:25:21.230179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.230164    4468 cpu_manager.go:414] "containerMap: removing container" podUID="53be671d-c05a-46ca-a810-6e08191611b8" containerName="revoke-keys"
Nov 01 06:25:21.230233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.230180    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53be671d-c05a-46ca-a810-6e08191611b8" containerName="revoke-keys"
Nov 01 06:25:21.230283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.230272    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="53be671d-c05a-46ca-a810-6e08191611b8" containerName="revoke-keys"
Nov 01 06:25:21.239181 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.239149    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.286449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.286414    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"]
Nov 01 06:25:21.330084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.330044    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.330304 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.330121    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9qr\" (UniqueName: \"kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.330304 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.330171    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.332869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.332839    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:25:21.333022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.332960    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:25:21.333175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.333156    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:25:21.341199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.341168    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:25:21.431158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.431123    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9qr\" (UniqueName: \"kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.431386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.431185    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.431386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.431268    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.431885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.431848    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.433807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.433784    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.487782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.487704    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9qr\" (UniqueName: \"kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr\") pod \"maas-api-revoke-sub-e305fb93-lnjgp\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.552639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.552609    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:21.937654 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:21.937619    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"]
Nov 01 06:25:21.937998 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:21.937973    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa07492e_b39d_49b3_bc47_ff259a427355.slice/crio-d1552009a85e4ee94ac6206d839a4c5dec550e7eee8bcf2f466a03f954628894 WatchSource:0}: Error finding container d1552009a85e4ee94ac6206d839a4c5dec550e7eee8bcf2f466a03f954628894: Status 404 returned error can't find the container with id d1552009a85e4ee94ac6206d839a4c5dec550e7eee8bcf2f466a03f954628894
Nov 01 06:25:22.334356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:22.334326    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp" event={"ID":"fa07492e-b39d-49b3-bc47-ff259a427355","Type":"ContainerStarted","Data":"d1552009a85e4ee94ac6206d839a4c5dec550e7eee8bcf2f466a03f954628894"}
Nov 01 06:25:23.342643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:23.342608    4468 generic.go:356] "Generic (PLEG): container finished" podID="fa07492e-b39d-49b3-bc47-ff259a427355" containerID="2afbb1b6572b872d89010f24570b2618b2064ab99d80a2d82c49a08351a6c900" exitCode=0
Nov 01 06:25:23.343053 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:23.342694    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp" event={"ID":"fa07492e-b39d-49b3-bc47-ff259a427355","Type":"ContainerDied","Data":"2afbb1b6572b872d89010f24570b2618b2064ab99d80a2d82c49a08351a6c900"}
Nov 01 06:25:24.501626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.501595    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:24.569708 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.569673    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr\" (UniqueName: \"kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr\") pod \"fa07492e-b39d-49b3-bc47-ff259a427355\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") "
Nov 01 06:25:24.569892 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.569770    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca\") pod \"fa07492e-b39d-49b3-bc47-ff259a427355\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") "
Nov 01 06:25:24.569892 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.569870    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca\") pod \"fa07492e-b39d-49b3-bc47-ff259a427355\" (UID: \"fa07492e-b39d-49b3-bc47-ff259a427355\") "
Nov 01 06:25:24.570269 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.570240    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca" pod "fa07492e-b39d-49b3-bc47-ff259a427355" (UID: "fa07492e-b39d-49b3-bc47-ff259a427355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:24.571987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.571962    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr" pod "fa07492e-b39d-49b3-bc47-ff259a427355" (UID: "fa07492e-b39d-49b3-bc47-ff259a427355"). InnerVolumeSpecName "kube-api-access-kv9qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:24.572087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.572061    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca" pod "fa07492e-b39d-49b3-bc47-ff259a427355" (UID: "fa07492e-b39d-49b3-bc47-ff259a427355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:25:24.672057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.671925    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:24.672057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.671958    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9qr\" (UniqueName: \"kubernetes.io/projected/fa07492e-b39d-49b3-bc47-ff259a427355-kube-api-access-kv9qr\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:24.672057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:24.671974    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa07492e-b39d-49b3-bc47-ff259a427355-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:25.357409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:25.357374    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp" event={"ID":"fa07492e-b39d-49b3-bc47-ff259a427355","Type":"ContainerDied","Data":"d1552009a85e4ee94ac6206d839a4c5dec550e7eee8bcf2f466a03f954628894"}
Nov 01 06:25:25.357607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:25.357421    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1552009a85e4ee94ac6206d839a4c5dec550e7eee8bcf2f466a03f954628894"
Nov 01 06:25:25.357607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:25.357422    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"
Nov 01 06:25:26.467826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.467786    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"]
Nov 01 06:25:26.468317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.468303    4468 cpu_manager.go:414] "containerMap: removing container" podUID="fa07492e-b39d-49b3-bc47-ff259a427355" containerName="revoke-keys"
Nov 01 06:25:26.468358 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.468318    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa07492e-b39d-49b3-bc47-ff259a427355" containerName="revoke-keys"
Nov 01 06:25:26.468406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.468398    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa07492e-b39d-49b3-bc47-ff259a427355" containerName="revoke-keys"
Nov 01 06:25:26.472036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.472015    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.492724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.492687    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.492911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.492742    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.492911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.492819    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.493017 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.492956    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.493094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.493075    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.493197 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.493180    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.493269 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.493255    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.493320 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.493285    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wz6\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.493387 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.493373    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.573440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.573390    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8ff537-openshift-default-dockercfg-sbjxb\""
Nov 01 06:25:26.578055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.578022    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"]
Nov 01 06:25:26.594258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594229    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594423 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594286    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594423 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594303    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wz6\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594437    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594535    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594587    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594619    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594681    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594731    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.594893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.594849    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.595213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.595056    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.595213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.595067    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.595213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.595120    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.595213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.595161    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.597262 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.597241    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.597346 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.597279    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.650776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.650743    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wz6\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.652305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.652285    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:26.785353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:26.785255    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:27.219943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.219913    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"]
Nov 01 06:25:27.221036 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:27.221000    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4382c7_7f1c_4274_914c_a2a106096b52.slice/crio-7e1502d4f120d847774f667522c15d6888e52f236ab06c475d0e8b8084c1d0b6 WatchSource:0}: Error finding container 7e1502d4f120d847774f667522c15d6888e52f236ab06c475d0e8b8084c1d0b6: Status 404 returned error can't find the container with id 7e1502d4f120d847774f667522c15d6888e52f236ab06c475d0e8b8084c1d0b6
Nov 01 06:25:27.223595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.223550    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:27.223704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.223643    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:27.223704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.223682    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:27.373007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.372968    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" event={"ID":"9a4382c7-7f1c-4274-914c-a2a106096b52","Type":"ContainerStarted","Data":"e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83"}
Nov 01 06:25:27.373158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.373033    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" event={"ID":"9a4382c7-7f1c-4274-914c-a2a106096b52","Type":"ContainerStarted","Data":"7e1502d4f120d847774f667522c15d6888e52f236ab06c475d0e8b8084c1d0b6"}
Nov 01 06:25:27.491980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.491862    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podStartSLOduration=1.491807019 podStartE2EDuration="1.491807019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:27.48945226 +0000 UTC m=+939.057259091" watchObservedRunningTime="2026-11-01 06:25:27.491807019 +0000 UTC m=+939.059613861"
Nov 01 06:25:27.786324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:27.786225    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:28.380743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.380707    4468 generic.go:356] "Generic (PLEG): container finished" podID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerID="d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1" exitCode=0
Nov 01 06:25:28.380963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.380775    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" event={"ID":"646fd28f-2ae4-48ac-931f-b3f12abad15e","Type":"ContainerDied","Data":"d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"}
Nov 01 06:25:28.381285 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.381270    4468 scope.go:122] "RemoveContainer" containerID="d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"
Nov 01 06:25:28.382590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.382562    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:28.382647 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.382628    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:28.382688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.382660    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:28.787014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.786924    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 01 06:25:28.787014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:28.786988    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 01 06:25:29.389302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.389108    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" event={"ID":"646fd28f-2ae4-48ac-931f-b3f12abad15e","Type":"ContainerStarted","Data":"d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7"}
Nov 01 06:25:29.452807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.452662    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:25:29.453053 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.453028    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:25:29.453159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.453086    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:25:29.632494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.629212    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"]
Nov 01 06:25:29.659541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.659445    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.667110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.667077    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"]
Nov 01 06:25:29.719994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.719946    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:25:29.747804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.720460    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:25:29.748077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.720537    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:25:29.748208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.721149    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:25:29.762410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.762375    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwg6g\" (UniqueName: \"kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.762738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.762723    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.762926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.762914    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.785677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.785648    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 01 06:25:29.785828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.785695    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 01 06:25:29.864285 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.864249    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwg6g\" (UniqueName: \"kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.864902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.864880    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.865145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.865125    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.865770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.865739    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.867646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.867614    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.942559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.938766    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwg6g\" (UniqueName: \"kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g\") pod \"maas-api-revoke-sub-e70751ef-9g4df\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:29.975357 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:29.975310    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:30.396124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:30.395893    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"]
Nov 01 06:25:30.398178 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:30.398151    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc027ca87_e45e_4cd3_ab6a_8f45daca0472.slice/crio-72cb0c52a398d26426b01894c5cf6550668fe7919e625f636d1e9f3ff738142a WatchSource:0}: Error finding container 72cb0c52a398d26426b01894c5cf6550668fe7919e625f636d1e9f3ff738142a: Status 404 returned error can't find the container with id 72cb0c52a398d26426b01894c5cf6550668fe7919e625f636d1e9f3ff738142a
Nov 01 06:25:30.451497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:30.450871    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:25:30.451497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:30.450929    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:25:30.785859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:30.785823    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 01 06:25:30.786038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:30.785879    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 01 06:25:31.445899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.444808    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df" event={"ID":"c027ca87-e45e-4cd3-ab6a-8f45daca0472","Type":"ContainerStarted","Data":"00b41ca9a3389cfd607b5f8e28a067b9f2576c22fc23e8cd3ca4494ac66f91a6"}
Nov 01 06:25:31.445899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.444877    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df" event={"ID":"c027ca87-e45e-4cd3-ab6a-8f45daca0472","Type":"ContainerStarted","Data":"72cb0c52a398d26426b01894c5cf6550668fe7919e625f636d1e9f3ff738142a"}
Nov 01 06:25:31.454997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.454969    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:25:31.455255 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.455229    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:25:31.560327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.560275    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df" podStartSLOduration=2.560261957 podStartE2EDuration="2.560261957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:31.557837359 +0000 UTC m=+943.125644190" watchObservedRunningTime="2026-11-01 06:25:31.560261957 +0000 UTC m=+943.128068785"
Nov 01 06:25:31.786503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.786381    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 01 06:25:31.786503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:31.786440    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 01 06:25:32.462522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:32.461729    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:25:32.462522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:32.461847    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:25:32.470569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:32.470534    4468 generic.go:356] "Generic (PLEG): container finished" podID="c027ca87-e45e-4cd3-ab6a-8f45daca0472" containerID="00b41ca9a3389cfd607b5f8e28a067b9f2576c22fc23e8cd3ca4494ac66f91a6" exitCode=0
Nov 01 06:25:32.470740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:32.470708    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df" event={"ID":"c027ca87-e45e-4cd3-ab6a-8f45daca0472","Type":"ContainerDied","Data":"00b41ca9a3389cfd607b5f8e28a067b9f2576c22fc23e8cd3ca4494ac66f91a6"}
Nov 01 06:25:32.786106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:32.786008    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 01 06:25:32.786106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:32.786074    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 01 06:25:32.973688 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:25:32.973652    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:25:33.450046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.450004    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:25:33.450232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.450068    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:25:33.667182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.667150    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:33.757166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.756858    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g\" (UniqueName: \"kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g\") pod \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") "
Nov 01 06:25:33.757166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.756978    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca\") pod \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") "
Nov 01 06:25:33.757397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.757229    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca\") pod \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\" (UID: \"c027ca87-e45e-4cd3-ab6a-8f45daca0472\") "
Nov 01 06:25:33.758047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.758021    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca" pod "c027ca87-e45e-4cd3-ab6a-8f45daca0472" (UID: "c027ca87-e45e-4cd3-ab6a-8f45daca0472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:33.759329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.759307    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g" pod "c027ca87-e45e-4cd3-ab6a-8f45daca0472" (UID: "c027ca87-e45e-4cd3-ab6a-8f45daca0472"). InnerVolumeSpecName "kube-api-access-pwg6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:33.759774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.759756    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca" pod "c027ca87-e45e-4cd3-ab6a-8f45daca0472" (UID: "c027ca87-e45e-4cd3-ab6a-8f45daca0472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:25:33.786375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.786328    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 01 06:25:33.786578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.786554    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 01 06:25:33.859208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.859167    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:33.859208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.859212    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwg6g\" (UniqueName: \"kubernetes.io/projected/c027ca87-e45e-4cd3-ab6a-8f45daca0472-kube-api-access-pwg6g\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:33.859425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:33.859226    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c027ca87-e45e-4cd3-ab6a-8f45daca0472-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:34.485129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.485092    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"
Nov 01 06:25:34.485129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.485121    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df" event={"ID":"c027ca87-e45e-4cd3-ab6a-8f45daca0472","Type":"ContainerDied","Data":"72cb0c52a398d26426b01894c5cf6550668fe7919e625f636d1e9f3ff738142a"}
Nov 01 06:25:34.485406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.485170    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72cb0c52a398d26426b01894c5cf6550668fe7919e625f636d1e9f3ff738142a"
Nov 01 06:25:34.499203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.499173    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:25:34.499504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.499453    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:25:34.500743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.500726    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 06:25:34.790188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.790112    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:34.790687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.790506    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:34.791349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:34.791332    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:25:37.915566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:37.915533    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"]
Nov 01 06:25:37.916211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:37.916194    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c027ca87-e45e-4cd3-ab6a-8f45daca0472" containerName="revoke-keys"
Nov 01 06:25:37.916253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:37.916214    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c027ca87-e45e-4cd3-ab6a-8f45daca0472" containerName="revoke-keys"
Nov 01 06:25:37.916311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:37.916303    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c027ca87-e45e-4cd3-ab6a-8f45daca0472" containerName="revoke-keys"
Nov 01 06:25:37.919629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:37.919613    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.012748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.012702    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.012918 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.012813    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.012964 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.012956    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lkb\" (UniqueName: \"kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.022170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.022143    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:25:38.022170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.022173    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:25:38.022376 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.022175    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:25:38.022376 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.022217    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:25:38.025589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.025572    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"]
Nov 01 06:25:38.114379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.114343    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lkb\" (UniqueName: \"kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.114589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.114446    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.114589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.114510    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.115175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.115148    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.117045 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.117018    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.170726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.170651    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lkb\" (UniqueName: \"kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb\") pod \"maas-api-revoke-sub-d9ca83c8-l8rlv\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.232522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.232491    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:38.605498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:38.605455    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"]
Nov 01 06:25:38.607223 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:38.607195    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da4b7c_ac75_41f2_9873_324abdb5519d.slice/crio-59ce419c59c8fe78a071e1c2f94a633bde87adf57aa16238dd2c33800b28b8c3 WatchSource:0}: Error finding container 59ce419c59c8fe78a071e1c2f94a633bde87adf57aa16238dd2c33800b28b8c3: Status 404 returned error can't find the container with id 59ce419c59c8fe78a071e1c2f94a633bde87adf57aa16238dd2c33800b28b8c3
Nov 01 06:25:39.518481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:39.518442    4468 generic.go:356] "Generic (PLEG): container finished" podID="89da4b7c-ac75-41f2-9873-324abdb5519d" containerID="1eb8e233287d534872d9ca8f3d1688c61824cdebe2722383bb6f20d5a6ea435a" exitCode=0
Nov 01 06:25:39.518869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:39.518507    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv" event={"ID":"89da4b7c-ac75-41f2-9873-324abdb5519d","Type":"ContainerDied","Data":"1eb8e233287d534872d9ca8f3d1688c61824cdebe2722383bb6f20d5a6ea435a"}
Nov 01 06:25:39.518869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:39.518561    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv" event={"ID":"89da4b7c-ac75-41f2-9873-324abdb5519d","Type":"ContainerStarted","Data":"59ce419c59c8fe78a071e1c2f94a633bde87adf57aa16238dd2c33800b28b8c3"}
Nov 01 06:25:40.658863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.658839    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:40.846056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.846019    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca\") pod \"89da4b7c-ac75-41f2-9873-324abdb5519d\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") "
Nov 01 06:25:40.846238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.846175    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca\") pod \"89da4b7c-ac75-41f2-9873-324abdb5519d\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") "
Nov 01 06:25:40.846238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.846212    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb\" (UniqueName: \"kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb\") pod \"89da4b7c-ac75-41f2-9873-324abdb5519d\" (UID: \"89da4b7c-ac75-41f2-9873-324abdb5519d\") "
Nov 01 06:25:40.846645 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.846620    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca" pod "89da4b7c-ac75-41f2-9873-324abdb5519d" (UID: "89da4b7c-ac75-41f2-9873-324abdb5519d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:40.846813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.846798    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:40.848297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.848277    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca" pod "89da4b7c-ac75-41f2-9873-324abdb5519d" (UID: "89da4b7c-ac75-41f2-9873-324abdb5519d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:25:40.848457 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.848443    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb" pod "89da4b7c-ac75-41f2-9873-324abdb5519d" (UID: "89da4b7c-ac75-41f2-9873-324abdb5519d"). InnerVolumeSpecName "kube-api-access-49lkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:40.947810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.947776    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89da4b7c-ac75-41f2-9873-324abdb5519d-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:40.947810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:40.947804    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49lkb\" (UniqueName: \"kubernetes.io/projected/89da4b7c-ac75-41f2-9873-324abdb5519d-kube-api-access-49lkb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:41.531780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:41.531744    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv" event={"ID":"89da4b7c-ac75-41f2-9873-324abdb5519d","Type":"ContainerDied","Data":"59ce419c59c8fe78a071e1c2f94a633bde87adf57aa16238dd2c33800b28b8c3"}
Nov 01 06:25:41.531780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:41.531775    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"
Nov 01 06:25:41.531997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:41.531820    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ce419c59c8fe78a071e1c2f94a633bde87adf57aa16238dd2c33800b28b8c3"
Nov 01 06:25:43.705536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.705504    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"]
Nov 01 06:25:43.706025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.706013    4468 cpu_manager.go:414] "containerMap: removing container" podUID="89da4b7c-ac75-41f2-9873-324abdb5519d" containerName="revoke-keys"
Nov 01 06:25:43.706068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.706027    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89da4b7c-ac75-41f2-9873-324abdb5519d" containerName="revoke-keys"
Nov 01 06:25:43.706127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.706116    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="89da4b7c-ac75-41f2-9873-324abdb5519d" containerName="revoke-keys"
Nov 01 06:25:43.709268 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.709250    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.779627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.779587    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bd9d24-openshift-default-dockercfg-m7dgp\""
Nov 01 06:25:43.786221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786194    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786227    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786259    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786416    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpc98\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786576    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786677    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786709    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.786823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786817    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.787060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.786848    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.814193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.814163    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"]
Nov 01 06:25:43.887861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.887819    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888073 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.887895    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpc98\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888073 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.887936    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888073 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888014    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888073 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888044    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888081    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888113    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888224    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888252    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888722    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888745    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.888953 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888866    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.889015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.888999    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.889484 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.889442    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.891837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.891801    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.892122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.892096    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.945175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.945144    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:43.945376 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:43.945351    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpc98\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98\") pod \"e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") " pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:44.023017 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.022931    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:44.450315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.450286    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"]
Nov 01 06:25:44.451966 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:44.451935    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fa8b9a_2e39_4c44_9d6e_e72073a42b6c.slice/crio-e22c5092ce56b15a61c1897afe698043373ff8f04ff76cbdba3e343f775266f6 WatchSource:0}: Error finding container e22c5092ce56b15a61c1897afe698043373ff8f04ff76cbdba3e343f775266f6: Status 404 returned error can't find the container with id e22c5092ce56b15a61c1897afe698043373ff8f04ff76cbdba3e343f775266f6
Nov 01 06:25:44.454308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.454273    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:44.454390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.454342    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:44.454390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.454370    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:44.601791 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.601749    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" event={"ID":"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c","Type":"ContainerStarted","Data":"498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"}
Nov 01 06:25:44.601962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.601825    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" event={"ID":"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c","Type":"ContainerStarted","Data":"e22c5092ce56b15a61c1897afe698043373ff8f04ff76cbdba3e343f775266f6"}
Nov 01 06:25:44.698426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:44.698307    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podStartSLOduration=1.698287368 podStartE2EDuration="1.698287368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:44.69670027 +0000 UTC m=+956.264507117" watchObservedRunningTime="2026-11-01 06:25:44.698287368 +0000 UTC m=+956.266094199"
Nov 01 06:25:45.023910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:45.023828    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:45.029229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:45.029200    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:45.589503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:45.589435    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:45.590551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:45.590528    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:25:53.155491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:53.155314    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:25:56.972840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:56.972799    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"]
Nov 01 06:25:56.977949 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:56.977928    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.032852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.032805    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:25:57.035041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.035010    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"]
Nov 01 06:25:57.055344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.055301    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.055561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.055542    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6h4\" (UniqueName: \"kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.055655 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.055604    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.080094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.080063    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:25:57.080094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.080075    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:25:57.080322 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.080221    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:25:57.157146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.157113    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6h4\" (UniqueName: \"kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.157351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.157177    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.157351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.157339    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.158075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.157995    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.160216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.160191    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.216556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.216522    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6h4\" (UniqueName: \"kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4\") pod \"maas-api-revoke-sub-1cb4b916-lf8w6\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.299489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.295667    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:57.427257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.426590    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"]
Nov 01 06:25:57.434585 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.434135    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.488558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.488372    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"]
Nov 01 06:25:57.567760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.567669    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.568044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.568024    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.568267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.568245    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.568417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.568398    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.568579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.568560    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.568808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.568786    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdm9\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.569198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.569165    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.569411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.569386    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.569571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.569487    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.670699 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.670658    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdm9\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.670911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.670714    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.670911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.670777    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.670911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.670846    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.670916    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671017    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671045    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671073    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671106    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671223    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671312    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671882    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.671914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.671907    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.672380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.672355    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.673898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.673870    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.673996 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.673907    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.682639 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:57.682597    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853d32de_15c6_497d_92ab_646db5f878cd.slice/crio-930578291c92a44ffc233b6753d6bee93f01e269f4f10a68a209a026d2bdcdd9 WatchSource:0}: Error finding container 930578291c92a44ffc233b6753d6bee93f01e269f4f10a68a209a026d2bdcdd9: Status 404 returned error can't find the container with id 930578291c92a44ffc233b6753d6bee93f01e269f4f10a68a209a026d2bdcdd9
Nov 01 06:25:57.684629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.684604    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"]
Nov 01 06:25:57.708975 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.708941    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6" event={"ID":"853d32de-15c6-497d-92ab-646db5f878cd","Type":"ContainerStarted","Data":"930578291c92a44ffc233b6753d6bee93f01e269f4f10a68a209a026d2bdcdd9"}
Nov 01 06:25:57.727860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.727830    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.728030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.727997    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdm9\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:57.751697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:57.751657    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:58.255915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.255880    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"]
Nov 01 06:25:58.257021 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:25:58.256977    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1571f43b_fc62_4861_b609_16c62203a1f1.slice/crio-6a1e213a58edd7efbe1aba1d113745e1d703adc6a76936aadc36586fae5c5387 WatchSource:0}: Error finding container 6a1e213a58edd7efbe1aba1d113745e1d703adc6a76936aadc36586fae5c5387: Status 404 returned error can't find the container with id 6a1e213a58edd7efbe1aba1d113745e1d703adc6a76936aadc36586fae5c5387
Nov 01 06:25:58.259868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.259827    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:58.259949 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.259917    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:58.259988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.259970    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:25:58.717150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.717112    4468 generic.go:356] "Generic (PLEG): container finished" podID="853d32de-15c6-497d-92ab-646db5f878cd" containerID="db637eba3130ac946861d96baae7731b62e8beb18e62604f6430d78b3da03e90" exitCode=0
Nov 01 06:25:58.717337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.717223    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6" event={"ID":"853d32de-15c6-497d-92ab-646db5f878cd","Type":"ContainerDied","Data":"db637eba3130ac946861d96baae7731b62e8beb18e62604f6430d78b3da03e90"}
Nov 01 06:25:58.718965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.718936    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" event={"ID":"1571f43b-fc62-4861-b609-16c62203a1f1","Type":"ContainerStarted","Data":"a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6"}
Nov 01 06:25:58.719084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.718982    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" event={"ID":"1571f43b-fc62-4861-b609-16c62203a1f1","Type":"ContainerStarted","Data":"6a1e213a58edd7efbe1aba1d113745e1d703adc6a76936aadc36586fae5c5387"}
Nov 01 06:25:58.752145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.752110    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:25:58.948596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:58.948488    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podStartSLOduration=1.948460018 podStartE2EDuration="1.948460018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:58.947686655 +0000 UTC m=+970.515493508" watchObservedRunningTime="2026-11-01 06:25:58.948460018 +0000 UTC m=+970.516266848"
Nov 01 06:25:59.754293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.754257    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 06:25:59.754710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.754339    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 06:25:59.895547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.895504    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:25:59.947805 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.947778    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca\") pod \"853d32de-15c6-497d-92ab-646db5f878cd\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") "
Nov 01 06:25:59.947994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.947864    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca\") pod \"853d32de-15c6-497d-92ab-646db5f878cd\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") "
Nov 01 06:25:59.947994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.947903    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4\" (UniqueName: \"kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4\") pod \"853d32de-15c6-497d-92ab-646db5f878cd\" (UID: \"853d32de-15c6-497d-92ab-646db5f878cd\") "
Nov 01 06:25:59.948214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.948182    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca" pod "853d32de-15c6-497d-92ab-646db5f878cd" (UID: "853d32de-15c6-497d-92ab-646db5f878cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:59.950078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.950050    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca" pod "853d32de-15c6-497d-92ab-646db5f878cd" (UID: "853d32de-15c6-497d-92ab-646db5f878cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:25:59.950179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:25:59.950088    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4" pod "853d32de-15c6-497d-92ab-646db5f878cd" (UID: "853d32de-15c6-497d-92ab-646db5f878cd"). InnerVolumeSpecName "kube-api-access-sj6h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:00.049818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.049784    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:00.049818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.049820    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853d32de-15c6-497d-92ab-646db5f878cd-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:00.050081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.049830    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6h4\" (UniqueName: \"kubernetes.io/projected/853d32de-15c6-497d-92ab-646db5f878cd-kube-api-access-sj6h4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:00.078316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.078273    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"]
Nov 01 06:26:00.081253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.081225    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-507e5841-z2htc"]
Nov 01 06:26:00.590288 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.590253    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184e1f7c-ffd7-4273-841d-b46ba0c3a131" path="/var/lib/kubelet/pods/184e1f7c-ffd7-4273-841d-b46ba0c3a131/volumes"
Nov 01 06:26:00.742753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.742722    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"
Nov 01 06:26:00.742906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.742756    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6" event={"ID":"853d32de-15c6-497d-92ab-646db5f878cd","Type":"ContainerDied","Data":"930578291c92a44ffc233b6753d6bee93f01e269f4f10a68a209a026d2bdcdd9"}
Nov 01 06:26:00.742906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.742802    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="930578291c92a44ffc233b6753d6bee93f01e269f4f10a68a209a026d2bdcdd9"
Nov 01 06:26:00.752478 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.752437    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 06:26:00.752609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:00.752499    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 06:26:01.752457 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:01.752424    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 06:26:01.752457 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:01.752490    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 06:26:02.753225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:02.753194    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 06:26:02.753670 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:02.753247    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 06:26:02.956720 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:26:02.956679    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:26:03.752434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:03.752398    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 06:26:03.752604 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:03.752487    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 06:26:04.752976 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:04.752932    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 06:26:04.753409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:04.752999    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 06:26:05.756910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:05.756882    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:26:05.757326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:05.757267    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:26:05.757863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:05.757845    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:26:07.763403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.763367    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"]
Nov 01 06:26:07.764052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.764035    4468 cpu_manager.go:414] "containerMap: removing container" podUID="853d32de-15c6-497d-92ab-646db5f878cd" containerName="revoke-keys"
Nov 01 06:26:07.764097 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.764055    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853d32de-15c6-497d-92ab-646db5f878cd" containerName="revoke-keys"
Nov 01 06:26:07.764173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.764163    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="853d32de-15c6-497d-92ab-646db5f878cd" containerName="revoke-keys"
Nov 01 06:26:07.768790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.768749    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.844722 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.843685    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.844722 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.843836    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.844722 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.843970    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pwv\" (UniqueName: \"kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.846698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.846662    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"]
Nov 01 06:26:07.879934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.879902    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:26:07.880178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.880155    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:26:07.880275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.880029    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:26:07.880275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.879900    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:26:07.945550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.945509    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.945731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.945668    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.945807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.945789    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pwv\" (UniqueName: \"kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.946399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.946322    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:07.948426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:07.948403    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:08.002502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:08.002444    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pwv\" (UniqueName: \"kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv\") pod \"maas-api-revoke-sub-e980b43b-84cb8\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:08.108539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:08.108490    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:08.565313 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:08.564228    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"]
Nov 01 06:26:08.877492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:08.877062    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8" event={"ID":"09f2bcee-8b32-4719-8b35-b1cab94d1f07","Type":"ContainerStarted","Data":"555c87cffaf640ee17e213e1b80c7c4bd0fc433a72abc2510b9d2f24448ae882"}
Nov 01 06:26:09.885498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:09.885365    4468 generic.go:356] "Generic (PLEG): container finished" podID="09f2bcee-8b32-4719-8b35-b1cab94d1f07" containerID="675bdfc4c4e6687feec0e6ad056b9c4e1827a8600aa2749020f9235c54b989a7" exitCode=0
Nov 01 06:26:09.885498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:09.885449    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8" event={"ID":"09f2bcee-8b32-4719-8b35-b1cab94d1f07","Type":"ContainerDied","Data":"675bdfc4c4e6687feec0e6ad056b9c4e1827a8600aa2749020f9235c54b989a7"}
Nov 01 06:26:11.051667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.051636    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:11.110127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.110091    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv\" (UniqueName: \"kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv\") pod \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") "
Nov 01 06:26:11.110301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.110164    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca\") pod \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") "
Nov 01 06:26:11.110301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.110267    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca\") pod \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\" (UID: \"09f2bcee-8b32-4719-8b35-b1cab94d1f07\") "
Nov 01 06:26:11.110566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.110540    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca" pod "09f2bcee-8b32-4719-8b35-b1cab94d1f07" (UID: "09f2bcee-8b32-4719-8b35-b1cab94d1f07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:11.110737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.110719    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:11.112480 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.112447    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca" pod "09f2bcee-8b32-4719-8b35-b1cab94d1f07" (UID: "09f2bcee-8b32-4719-8b35-b1cab94d1f07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:11.112538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.112527    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv" pod "09f2bcee-8b32-4719-8b35-b1cab94d1f07" (UID: "09f2bcee-8b32-4719-8b35-b1cab94d1f07"). InnerVolumeSpecName "kube-api-access-b6pwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:11.212295 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.212211    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pwv\" (UniqueName: \"kubernetes.io/projected/09f2bcee-8b32-4719-8b35-b1cab94d1f07-kube-api-access-b6pwv\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:11.212295 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.212242    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2bcee-8b32-4719-8b35-b1cab94d1f07-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:11.899768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.899728    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8" event={"ID":"09f2bcee-8b32-4719-8b35-b1cab94d1f07","Type":"ContainerDied","Data":"555c87cffaf640ee17e213e1b80c7c4bd0fc433a72abc2510b9d2f24448ae882"}
Nov 01 06:26:11.899952 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.899772    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"
Nov 01 06:26:11.899952 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:11.899821    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555c87cffaf640ee17e213e1b80c7c4bd0fc433a72abc2510b9d2f24448ae882"
Nov 01 06:26:17.722083 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:17.722046    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:26:24.050573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.050534    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"]
Nov 01 06:26:24.051082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.051068    4468 cpu_manager.go:414] "containerMap: removing container" podUID="09f2bcee-8b32-4719-8b35-b1cab94d1f07" containerName="revoke-keys"
Nov 01 06:26:24.051126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.051083    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f2bcee-8b32-4719-8b35-b1cab94d1f07" containerName="revoke-keys"
Nov 01 06:26:24.051176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.051168    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f2bcee-8b32-4719-8b35-b1cab94d1f07" containerName="revoke-keys"
Nov 01 06:26:24.054862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.054845    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.084318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.084285    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.084566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.084348    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5tf\" (UniqueName: \"kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.084566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.084403    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.107797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.107703    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:26:24.108000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.107851    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:26:24.157360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.157318    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:26:24.157360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.157325    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:26:24.161180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.161158    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"]
Nov 01 06:26:24.185385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.185346    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.185596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.185405    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5tf\" (UniqueName: \"kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.185596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.185452    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.186244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.186221    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.188203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.188178    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.243807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.243778    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5tf\" (UniqueName: \"kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf\") pod \"maas-api-revoke-sub-b3231be4-2xh2s\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.369012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.368925    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:24.743933 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:26:24.743901    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a65f9e_aaab_4d0b_9253_59a7f4108227.slice/crio-2396d290688b3442fb0597061665dda0309eb27216de42a6391c0834a89ff7ef WatchSource:0}: Error finding container 2396d290688b3442fb0597061665dda0309eb27216de42a6391c0834a89ff7ef: Status 404 returned error can't find the container with id 2396d290688b3442fb0597061665dda0309eb27216de42a6391c0834a89ff7ef
Nov 01 06:26:24.745451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.745424    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"]
Nov 01 06:26:24.989665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:24.989630    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s" event={"ID":"84a65f9e-aaab-4d0b-9253-59a7f4108227","Type":"ContainerStarted","Data":"2396d290688b3442fb0597061665dda0309eb27216de42a6391c0834a89ff7ef"}
Nov 01 06:26:26.003818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:26.002326    4468 generic.go:356] "Generic (PLEG): container finished" podID="84a65f9e-aaab-4d0b-9253-59a7f4108227" containerID="1e2c6e8cc9c9f28779bc3bead20738f67ecf2d8ada0bac3d7c24108626339a90" exitCode=0
Nov 01 06:26:26.003818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:26.002441    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s" event={"ID":"84a65f9e-aaab-4d0b-9253-59a7f4108227","Type":"ContainerDied","Data":"1e2c6e8cc9c9f28779bc3bead20738f67ecf2d8ada0bac3d7c24108626339a90"}
Nov 01 06:26:27.165309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.165281    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:27.221702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.221665    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca\") pod \"84a65f9e-aaab-4d0b-9253-59a7f4108227\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") "
Nov 01 06:26:27.221877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.221725    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf\" (UniqueName: \"kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf\") pod \"84a65f9e-aaab-4d0b-9253-59a7f4108227\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") "
Nov 01 06:26:27.221877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.221793    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca\") pod \"84a65f9e-aaab-4d0b-9253-59a7f4108227\" (UID: \"84a65f9e-aaab-4d0b-9253-59a7f4108227\") "
Nov 01 06:26:27.222158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.222137    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca" pod "84a65f9e-aaab-4d0b-9253-59a7f4108227" (UID: "84a65f9e-aaab-4d0b-9253-59a7f4108227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:27.223956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.223927    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca" pod "84a65f9e-aaab-4d0b-9253-59a7f4108227" (UID: "84a65f9e-aaab-4d0b-9253-59a7f4108227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:27.224116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.224089    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf" pod "84a65f9e-aaab-4d0b-9253-59a7f4108227" (UID: "84a65f9e-aaab-4d0b-9253-59a7f4108227"). InnerVolumeSpecName "kube-api-access-8t5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:27.323212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.323169    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:27.323212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.323213    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a65f9e-aaab-4d0b-9253-59a7f4108227-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:27.323429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:27.323229    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5tf\" (UniqueName: \"kubernetes.io/projected/84a65f9e-aaab-4d0b-9253-59a7f4108227-kube-api-access-8t5tf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:28.016212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:28.016176    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"
Nov 01 06:26:28.016212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:28.016194    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s" event={"ID":"84a65f9e-aaab-4d0b-9253-59a7f4108227","Type":"ContainerDied","Data":"2396d290688b3442fb0597061665dda0309eb27216de42a6391c0834a89ff7ef"}
Nov 01 06:26:28.016496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:28.016279    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2396d290688b3442fb0597061665dda0309eb27216de42a6391c0834a89ff7ef"
Nov 01 06:26:31.930380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:31.930349    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"]
Nov 01 06:26:31.930937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:31.930725    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy" containerID="cri-o://706a069381ba00d1308450cc9f192f6f8dcaccb0fb9b907b402bd8b64001d4e7" gracePeriod=30
Nov 01 06:26:31.933783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:31.933758    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"]
Nov 01 06:26:31.933959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:31.933938    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy" containerID="cri-o://deb420d2b7bc4b4b9c08408a091aa49c1dfa9bdfb7c182da90daeeb77bcb30cc" gracePeriod=30
Nov 01 06:26:32.947481 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:26:32.947423    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:26:34.395609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.395575    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"]
Nov 01 06:26:34.396306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.396284    4468 cpu_manager.go:414] "containerMap: removing container" podUID="84a65f9e-aaab-4d0b-9253-59a7f4108227" containerName="revoke-keys"
Nov 01 06:26:34.396407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.396309    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a65f9e-aaab-4d0b-9253-59a7f4108227" containerName="revoke-keys"
Nov 01 06:26:34.396532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.396516    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a65f9e-aaab-4d0b-9253-59a7f4108227" containerName="revoke-keys"
Nov 01 06:26:34.399941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.399921    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.456301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.456265    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"]
Nov 01 06:26:34.502821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.502792    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:26:34.502821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.502797    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:26:34.503064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.502940    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:26:34.503293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.503271    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:26:34.518096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.518064    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.518292 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.518272    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.518358 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.518330    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lws\" (UniqueName: \"kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.619168 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.619133    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.619594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.619201    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lws\" (UniqueName: \"kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.619594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.619291    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.620318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.620297    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.649641 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.649567    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.700489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.690783    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lws\" (UniqueName: \"kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws\") pod \"maas-api-revoke-sub-d13c2146-kmg2m\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:34.715427 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:34.715395    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:35.100913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:35.100875    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"]
Nov 01 06:26:35.105402 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:26:35.105369    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41a81b9_6f64_42b5_afab_03e1eabcedb3.slice/crio-559fe846bfeaf19a6bae34264c5f46f602c7fa33a99fcb0000b1b1403abf698d WatchSource:0}: Error finding container 559fe846bfeaf19a6bae34264c5f46f602c7fa33a99fcb0000b1b1403abf698d: Status 404 returned error can't find the container with id 559fe846bfeaf19a6bae34264c5f46f602c7fa33a99fcb0000b1b1403abf698d
Nov 01 06:26:36.078360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:36.078329    4468 generic.go:356] "Generic (PLEG): container finished" podID="e41a81b9-6f64-42b5-afab-03e1eabcedb3" containerID="7cea31c20ab92255bd1649ec8251fb2bbc74d69b54270238457b9e4ce1307579" exitCode=0
Nov 01 06:26:36.078776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:36.078408    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m" event={"ID":"e41a81b9-6f64-42b5-afab-03e1eabcedb3","Type":"ContainerDied","Data":"7cea31c20ab92255bd1649ec8251fb2bbc74d69b54270238457b9e4ce1307579"}
Nov 01 06:26:36.078776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:36.078496    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m" event={"ID":"e41a81b9-6f64-42b5-afab-03e1eabcedb3","Type":"ContainerStarted","Data":"559fe846bfeaf19a6bae34264c5f46f602c7fa33a99fcb0000b1b1403abf698d"}
Nov 01 06:26:37.095477 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.095442    4468 generic.go:356] "Generic (PLEG): container finished" podID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerID="deb420d2b7bc4b4b9c08408a091aa49c1dfa9bdfb7c182da90daeeb77bcb30cc" exitCode=0
Nov 01 06:26:37.095983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.095537    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" event={"ID":"83242ea6-ab3a-4ac0-a37e-012b6c0b4265","Type":"ContainerDied","Data":"deb420d2b7bc4b4b9c08408a091aa49c1dfa9bdfb7c182da90daeeb77bcb30cc"}
Nov 01 06:26:37.098950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.098917    4468 generic.go:356] "Generic (PLEG): container finished" podID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerID="706a069381ba00d1308450cc9f192f6f8dcaccb0fb9b907b402bd8b64001d4e7" exitCode=0
Nov 01 06:26:37.099243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.099212    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" event={"ID":"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a","Type":"ContainerDied","Data":"706a069381ba00d1308450cc9f192f6f8dcaccb0fb9b907b402bd8b64001d4e7"}
Nov 01 06:26:37.226555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.226522    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:26:37.285447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.285425    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:26:37.289272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.289251    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:37.360306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360269    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.360306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360309    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.360582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360346    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca\") pod \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") "
Nov 01 06:26:37.360582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360376    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.360582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360410    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca\") pod \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") "
Nov 01 06:26:37.360582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360441    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.360582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360485    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.360582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360518    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.360894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360750    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.360894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360813    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.360894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360874    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.361052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360893    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360886    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.361052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360941    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.360972    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361027    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws\" (UniqueName: \"kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws\") pod \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\" (UID: \"e41a81b9-6f64-42b5-afab-03e1eabcedb3\") "
Nov 01 06:26:37.361311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361068    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361106    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca" pod "e41a81b9-6f64-42b5-afab-03e1eabcedb3" (UID: "e41a81b9-6f64-42b5-afab-03e1eabcedb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:37.361311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361138    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.361311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361174    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361228    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361289    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert\") pod \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\" (UID: \"83242ea6-ab3a-4ac0-a37e-012b6c0b4265\") "
Nov 01 06:26:37.361659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361372    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.361659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361483    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.361659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361511    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo\") pod \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\" (UID: \"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a\") "
Nov 01 06:26:37.362316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.362286    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.362424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.362321    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.362424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.362338    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.362424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.362353    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.361175    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363357    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca" pod "e41a81b9-6f64-42b5-afab-03e1eabcedb3" (UID: "e41a81b9-6f64-42b5-afab-03e1eabcedb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363428    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363973    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363589    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363601    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363654    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363682    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.364078    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363891    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.364138    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "kube-api-access-6zbxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.363940    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.364162    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:37.364177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.364246    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket" pod "83242ea6-ab3a-4ac0-a37e-012b6c0b4265" (UID: "83242ea6-ab3a-4ac0-a37e-012b6c0b4265"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:37.365141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.364673    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:26:37.365648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.365628    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws" pod "e41a81b9-6f64-42b5-afab-03e1eabcedb3" (UID: "e41a81b9-6f64-42b5-afab-03e1eabcedb3"). InnerVolumeSpecName "kube-api-access-v9lws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:37.365753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.365736    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft" pod "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" (UID: "8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a"). InnerVolumeSpecName "kube-api-access-prxft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464008    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464036    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464046    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464055    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9lws\" (UniqueName: \"kubernetes.io/projected/e41a81b9-6f64-42b5-afab-03e1eabcedb3-kube-api-access-v9lws\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464064    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464072    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464081    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbxt\" (UniqueName: \"kubernetes.io/projected/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-kube-api-access-6zbxt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464089    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464100    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464108    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxft\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-kube-api-access-prxft\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464116    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464125    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464133    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464141    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464150    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41a81b9-6f64-42b5-afab-03e1eabcedb3-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464160    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83242ea6-ab3a-4ac0-a37e-012b6c0b4265-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:37.464681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:37.464168    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:38.107230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.107199    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"
Nov 01 06:26:38.107732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.107234    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m" event={"ID":"e41a81b9-6f64-42b5-afab-03e1eabcedb3","Type":"ContainerDied","Data":"559fe846bfeaf19a6bae34264c5f46f602c7fa33a99fcb0000b1b1403abf698d"}
Nov 01 06:26:38.107732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.107281    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559fe846bfeaf19a6bae34264c5f46f602c7fa33a99fcb0000b1b1403abf698d"
Nov 01 06:26:38.108619 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.108587    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7" event={"ID":"83242ea6-ab3a-4ac0-a37e-012b6c0b4265","Type":"ContainerDied","Data":"59ad34e13e2c5e17dff1a8983f586c0cee00fe3a128d84b9706d82a77bf55b0f"}
Nov 01 06:26:38.108751 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.108594    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"
Nov 01 06:26:38.108751 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.108655    4468 scope.go:122] "RemoveContainer" containerID="deb420d2b7bc4b4b9c08408a091aa49c1dfa9bdfb7c182da90daeeb77bcb30cc"
Nov 01 06:26:38.110164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.110149    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"
Nov 01 06:26:38.110258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.110157    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln" event={"ID":"8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a","Type":"ContainerDied","Data":"560e20f3d46f25ede61331605ea519cb1d1afab2e6754b4e650aec55963c4094"}
Nov 01 06:26:38.120849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.120822    4468 scope.go:122] "RemoveContainer" containerID="706a069381ba00d1308450cc9f192f6f8dcaccb0fb9b907b402bd8b64001d4e7"
Nov 01 06:26:38.273799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.273707    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"]
Nov 01 06:26:38.323791 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.323760    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-nrzv7"]
Nov 01 06:26:38.431693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.431655    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"]
Nov 01 06:26:38.434830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.434803    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g6cln"]
Nov 01 06:26:38.591574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.591540    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" path="/var/lib/kubelet/pods/83242ea6-ab3a-4ac0-a37e-012b6c0b4265/volumes"
Nov 01 06:26:38.592008 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:38.591995    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" path="/var/lib/kubelet/pods/8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a/volumes"
Nov 01 06:26:39.083401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083365    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"]
Nov 01 06:26:39.083922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083909    4468 cpu_manager.go:414] "containerMap: removing container" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy"
Nov 01 06:26:39.083970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083924    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy"
Nov 01 06:26:39.083970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083952    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e41a81b9-6f64-42b5-afab-03e1eabcedb3" containerName="revoke-keys"
Nov 01 06:26:39.083970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083957    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41a81b9-6f64-42b5-afab-03e1eabcedb3" containerName="revoke-keys"
Nov 01 06:26:39.084062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083975    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy"
Nov 01 06:26:39.084062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.083980    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy"
Nov 01 06:26:39.084062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.084050    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1fdc09-8d2b-4c4e-bb72-3c4e15342f8a" containerName="istio-proxy"
Nov 01 06:26:39.084062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.084061    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="83242ea6-ab3a-4ac0-a37e-012b6c0b4265" containerName="istio-proxy"
Nov 01 06:26:39.084175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.084071    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41a81b9-6f64-42b5-afab-03e1eabcedb3" containerName="revoke-keys"
Nov 01 06:26:39.088852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.088830    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.141772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.141451    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:26:39.141772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.141607    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:26:39.141772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.141671    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:26:39.141772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.141741    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:26:39.143826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.143736    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"]
Nov 01 06:26:39.181503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.181446    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.181706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.181654    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.181706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.181693    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqn6\" (UniqueName: \"kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.283225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.283188    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.283225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.283234    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqn6\" (UniqueName: \"kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.283550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.283339    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.283986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.283965    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.285784 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.285757    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.339897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.339821    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqn6\" (UniqueName: \"kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6\") pod \"maas-api-revoke-sub-2a6e463f-m88lt\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.407013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.406972    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:39.784691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:39.784662    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"]
Nov 01 06:26:39.785903 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:26:39.785879    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ae4d0e_f57a_4aaa_8c6d_7e3c7c65d369.slice/crio-8caf42cf7f5a9c3c38c8a75e18c935e958695a8beb3791f336ab2e36893c6838 WatchSource:0}: Error finding container 8caf42cf7f5a9c3c38c8a75e18c935e958695a8beb3791f336ab2e36893c6838: Status 404 returned error can't find the container with id 8caf42cf7f5a9c3c38c8a75e18c935e958695a8beb3791f336ab2e36893c6838
Nov 01 06:26:40.139886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:40.139848    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt" event={"ID":"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369","Type":"ContainerStarted","Data":"2a2e21f30a4d26fc3e5e81c3769f24377c2f8762406b6b69d8297ccfe373bd49"}
Nov 01 06:26:40.140075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:40.139913    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt" event={"ID":"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369","Type":"ContainerStarted","Data":"8caf42cf7f5a9c3c38c8a75e18c935e958695a8beb3791f336ab2e36893c6838"}
Nov 01 06:26:40.252077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:40.251956    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt" podStartSLOduration=1.251924407 podStartE2EDuration="1.251924407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:40.251116045 +0000 UTC m=+1011.818922875" watchObservedRunningTime="2026-11-01 06:26:40.251924407 +0000 UTC m=+1011.819731238"
Nov 01 06:26:41.149206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:41.149167    4468 generic.go:356] "Generic (PLEG): container finished" podID="03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" containerID="2a2e21f30a4d26fc3e5e81c3769f24377c2f8762406b6b69d8297ccfe373bd49" exitCode=0
Nov 01 06:26:41.149369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:41.149238    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt" event={"ID":"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369","Type":"ContainerDied","Data":"2a2e21f30a4d26fc3e5e81c3769f24377c2f8762406b6b69d8297ccfe373bd49"}
Nov 01 06:26:42.313993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.313967    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:42.422802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.422769    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca\") pod \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") "
Nov 01 06:26:42.422956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.422828    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6\" (UniqueName: \"kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6\") pod \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") "
Nov 01 06:26:42.423077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.422971    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca\") pod \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\" (UID: \"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369\") "
Nov 01 06:26:42.423193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.423161    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca" pod "03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" (UID: "03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:42.423494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.423447    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:42.425061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.425040    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6" pod "03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" (UID: "03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369"). InnerVolumeSpecName "kube-api-access-gbqn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:42.425061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.425041    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca" pod "03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" (UID: "03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:42.524619 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.524583    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqn6\" (UniqueName: \"kubernetes.io/projected/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-kube-api-access-gbqn6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:42.524619 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.524615    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:42.734800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.734718    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"]
Nov 01 06:26:42.735307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.735293    4468 cpu_manager.go:414] "containerMap: removing container" podUID="03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" containerName="revoke-keys"
Nov 01 06:26:42.735346 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.735309    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" containerName="revoke-keys"
Nov 01 06:26:42.735403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.735394    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" containerName="revoke-keys"
Nov 01 06:26:42.738967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.738948    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.764923 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.764888    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"]
Nov 01 06:26:42.769178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.769153    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.814483 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.814432    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"]
Nov 01 06:26:42.828509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828458    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.828672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828539    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.828672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828601    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.828795 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828704    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.828795 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828744    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.828921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828805    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.828921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828856    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlm8c\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.829014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828925    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.829014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.828965    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.876116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.876077    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"]
Nov 01 06:26:42.930618 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930583    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.930618 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930623    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.930825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930640    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.930825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930655    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.930825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930742    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgf4f\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.930825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930789    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.930825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930814    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930861    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930897    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930930    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlm8c\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.930990    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931017    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931067    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931112    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931150    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931206    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931215    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931240    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931360    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931542    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:42.931612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931544    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931603    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.931701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.931627    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.933641 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.933610    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.933756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.933735    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.990919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.990849    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:42.991062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:42.990927    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlm8c\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:43.032430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032394    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.032629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032441    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.032629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032458    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.032629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032518    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.032629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032579    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.032629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032603    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032653    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032735    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032791    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgf4f\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.032909    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.033012    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.033274    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.033450    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.033753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.033730    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.034897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.034877    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.035021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.035004    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.053020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.052992    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:43.089663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.089630    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.089833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.089810    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgf4f\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.162597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.162563    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"
Nov 01 06:26:43.162787 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.162603    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt" event={"ID":"03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369","Type":"ContainerDied","Data":"8caf42cf7f5a9c3c38c8a75e18c935e958695a8beb3791f336ab2e36893c6838"}
Nov 01 06:26:43.162787 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.162687    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8caf42cf7f5a9c3c38c8a75e18c935e958695a8beb3791f336ab2e36893c6838"
Nov 01 06:26:43.383139 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.383098    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:43.482213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.482180    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"]
Nov 01 06:26:43.487170 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:26:43.487129    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadba30e_2774_4d82_ac07_efbf65db4acf.slice/crio-1cc64c0143884508846219ce5c19da6b458ce2b8e418a58f578052a63345dab5 WatchSource:0}: Error finding container 1cc64c0143884508846219ce5c19da6b458ce2b8e418a58f578052a63345dab5: Status 404 returned error can't find the container with id 1cc64c0143884508846219ce5c19da6b458ce2b8e418a58f578052a63345dab5
Nov 01 06:26:43.490391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.490348    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:43.490583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.490450    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:43.490583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.490544    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:43.763936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.763908    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"]
Nov 01 06:26:43.766568 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:26:43.766540    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838d3f5d_009d_4679_8a14_3347e50680e0.slice/crio-0f722946148cedc5d9496ad983c02a2914fcd584add77e66fbb9fde79486f96e WatchSource:0}: Error finding container 0f722946148cedc5d9496ad983c02a2914fcd584add77e66fbb9fde79486f96e: Status 404 returned error can't find the container with id 0f722946148cedc5d9496ad983c02a2914fcd584add77e66fbb9fde79486f96e
Nov 01 06:26:43.769094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.769057    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:43.769190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.769164    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:43.769261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:43.769214    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:44.171224 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.171188    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" event={"ID":"aadba30e-2774-4d82-ac07-efbf65db4acf","Type":"ContainerStarted","Data":"0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1"}
Nov 01 06:26:44.171409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.171255    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" event={"ID":"aadba30e-2774-4d82-ac07-efbf65db4acf","Type":"ContainerStarted","Data":"1cc64c0143884508846219ce5c19da6b458ce2b8e418a58f578052a63345dab5"}
Nov 01 06:26:44.173198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.173171    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" event={"ID":"838d3f5d-009d-4679-8a14-3347e50680e0","Type":"ContainerStarted","Data":"24a52f290364c95c098e732928a178002b994216538f62d8ca2f811920bafe6f"}
Nov 01 06:26:44.173381 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.173217    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" event={"ID":"838d3f5d-009d-4679-8a14-3347e50680e0","Type":"ContainerStarted","Data":"0f722946148cedc5d9496ad983c02a2914fcd584add77e66fbb9fde79486f96e"}
Nov 01 06:26:44.292595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.292533    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podStartSLOduration=2.292516701 podStartE2EDuration="2.292516701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:44.288213396 +0000 UTC m=+1015.856020261" watchObservedRunningTime="2026-11-01 06:26:44.292516701 +0000 UTC m=+1015.860323531"
Nov 01 06:26:44.384250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.384206    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:44.385871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.385845    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:44.386020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.385893    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:44.420504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:44.419385    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podStartSLOduration=2.419363609 podStartE2EDuration="2.419363609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:44.418848467 +0000 UTC m=+1015.986655298" watchObservedRunningTime="2026-11-01 06:26:44.419363609 +0000 UTC m=+1015.987170440"
Nov 01 06:26:45.054662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:45.054322    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:45.056017 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:45.055987    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:45.056134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:45.056035    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:45.384214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:45.384131    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:45.384214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:45.384188    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:46.053954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:46.053907    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:46.054119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:46.053975    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:46.389872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:46.384081    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:46.389872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:46.384137    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:47.053915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.053858    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:47.054136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.053937    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:47.383984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.383879    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:47.383984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.383944    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:47.457448 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.457411    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"]
Nov 01 06:26:47.465777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.465748    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.523491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.523336    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"]
Nov 01 06:26:47.566498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.549755    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:26:47.566498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.549829    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:26:47.566498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.549856    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:26:47.566498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.549949    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:26:47.578106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.578054    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:26:47.595819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.595773    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.596019 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.595882    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.596094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.596066    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8r2\" (UniqueName: \"kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.700014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.699898    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8r2\" (UniqueName: \"kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.700419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.700398    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.700677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.700661    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.709046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.708999    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.784525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.767949    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:47.786502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:47.786336    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8r2\" (UniqueName: \"kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2\") pod \"maas-api-revoke-sub-026954d6-txf9d\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:48.058501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:48.056874    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:48.058501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:48.056938    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:48.083024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:48.082971    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:48.443490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:48.435287    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:48.443490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:48.435383    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:49.086503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:49.066733    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:49.086503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:49.066818    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:49.462547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:49.459946    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"]
Nov 01 06:26:49.494401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:49.494367    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:49.571307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:49.570573    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:49.738883 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:49.738754    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d" event={"ID":"073364d1-fdbf-4f91-aae0-e90cfdd1f256","Type":"ContainerStarted","Data":"c37ddd05745c7c322568bb046157ee8169ba6e07dcdb91ac300d57d5f422170f"}
Nov 01 06:26:50.060440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.060247    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:50.060440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.060308    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:50.182499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.119567    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:26:50.331860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.331804    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:26:50.411495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.408012    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:50.434967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.434919    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:50.774804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.774765    4468 generic.go:356] "Generic (PLEG): container finished" podID="073364d1-fdbf-4f91-aae0-e90cfdd1f256" containerID="6626babce39d9714088fe3f998c47ef1c8e135bccf38181055c6f9d2edb95ba2" exitCode=0
Nov 01 06:26:50.775289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:50.775263    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d" event={"ID":"073364d1-fdbf-4f91-aae0-e90cfdd1f256","Type":"ContainerDied","Data":"6626babce39d9714088fe3f998c47ef1c8e135bccf38181055c6f9d2edb95ba2"}
Nov 01 06:26:51.053974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:51.053934    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 06:26:51.054154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:51.054004    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 06:26:51.384297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:51.384197    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 01 06:26:51.384297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:51.384262    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 01 06:26:52.058143 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.054758    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:52.189493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.185700    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca\") pod \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") "
Nov 01 06:26:52.189493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.185796    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca\") pod \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") "
Nov 01 06:26:52.189493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.185971    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2\" (UniqueName: \"kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2\") pod \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\" (UID: \"073364d1-fdbf-4f91-aae0-e90cfdd1f256\") "
Nov 01 06:26:52.195693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.194994    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca" pod "073364d1-fdbf-4f91-aae0-e90cfdd1f256" (UID: "073364d1-fdbf-4f91-aae0-e90cfdd1f256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:52.262317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.262282    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.266430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.266394    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2" pod "073364d1-fdbf-4f91-aae0-e90cfdd1f256" (UID: "073364d1-fdbf-4f91-aae0-e90cfdd1f256"). InnerVolumeSpecName "kube-api-access-4j8r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:52.266778 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.266742    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca" pod "073364d1-fdbf-4f91-aae0-e90cfdd1f256" (UID: "073364d1-fdbf-4f91-aae0-e90cfdd1f256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:52.364211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.364098    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073364d1-fdbf-4f91-aae0-e90cfdd1f256-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.364211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.364142    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8r2\" (UniqueName: \"kubernetes.io/projected/073364d1-fdbf-4f91-aae0-e90cfdd1f256-kube-api-access-4j8r2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.540929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.540889    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:52.542246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.542224    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:52.543418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.543395    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:26:52.634779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.633578    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:52.634879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.634818    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:52.639703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.639669    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 06:26:52.802546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.796842    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d" event={"ID":"073364d1-fdbf-4f91-aae0-e90cfdd1f256","Type":"ContainerDied","Data":"c37ddd05745c7c322568bb046157ee8169ba6e07dcdb91ac300d57d5f422170f"}
Nov 01 06:26:52.802546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.796910    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37ddd05745c7c322568bb046157ee8169ba6e07dcdb91ac300d57d5f422170f"
Nov 01 06:26:52.802546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:52.797821    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"
Nov 01 06:26:58.870863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:58.870761    4468 generic.go:356] "Generic (PLEG): container finished" podID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerID="498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a" exitCode=0
Nov 01 06:26:58.870863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:58.870831    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" event={"ID":"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c","Type":"ContainerDied","Data":"498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"}
Nov 01 06:26:58.871321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:58.871273    4468 scope.go:122] "RemoveContainer" containerID="498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"
Nov 01 06:26:58.872802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:58.872773    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:58.872876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:58.872844    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:58.872911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:58.872876    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:26:59.879868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:26:59.879833    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" event={"ID":"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c","Type":"ContainerStarted","Data":"e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62"}
Nov 01 06:27:00.027659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:00.027618    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:27:00.029035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:00.029005    4468 patch_prober.go:29] interesting pod/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 06:27:00.029165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:00.029061    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 06:27:00.590370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:00.590344    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:27:01.036492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:01.031226    4468 patch_prober.go:29] interesting pod/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 06:27:01.036492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:01.031287    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 06:27:02.027494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:02.026903    4468 patch_prober.go:29] interesting pod/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 06:27:02.027494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:02.026981    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 06:27:02.953795 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:27:02.953754    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:27:03.025494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:03.023888    4468 patch_prober.go:29] interesting pod/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 06:27:03.025494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:03.023949    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 06:27:04.024492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:04.023930    4468 patch_prober.go:29] interesting pod/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 06:27:04.024492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:04.023993    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 06:27:05.024342 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:05.024291    4468 patch_prober.go:29] interesting pod/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 06:27:05.024605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:05.024358    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 06:27:06.031373 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:06.031341    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:27:06.035033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:06.035004    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:27:09.219786 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:09.219744    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"]
Nov 01 06:27:12.277695 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.277663    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"]
Nov 01 06:27:12.278252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.278237    4468 cpu_manager.go:414] "containerMap: removing container" podUID="073364d1-fdbf-4f91-aae0-e90cfdd1f256" containerName="revoke-keys"
Nov 01 06:27:12.278296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.278254    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073364d1-fdbf-4f91-aae0-e90cfdd1f256" containerName="revoke-keys"
Nov 01 06:27:12.278348 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.278339    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="073364d1-fdbf-4f91-aae0-e90cfdd1f256" containerName="revoke-keys"
Nov 01 06:27:12.281913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.281898    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.354965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.354930    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.354978    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9jp\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355000    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355079    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355255 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355212    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355263    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355406    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355533    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.355616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.355600    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.388987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.388955    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"]
Nov 01 06:27:12.456505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456446    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456512    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456547    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456575    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9jp\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456595    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456609    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456654    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456989 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456715    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456989 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456806    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.456989 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.456894    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.457198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.457170    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.457311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.457213    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.457311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.457280    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.457311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.457288    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.459238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.459213    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.459316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.459305    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.512971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.512947    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.513212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.513192    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9jp\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:12.595912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:12.595883    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:13.021904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:13.021872    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"]
Nov 01 06:27:13.022172 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:27:13.022152    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e777d1c_f8aa_43e3_9263_670757d9239f.slice/crio-f702cec9255f1199582b8f8f5734df8b0d0d9be9f21060e4b5a44cf8003ae0ff WatchSource:0}: Error finding container f702cec9255f1199582b8f8f5734df8b0d0d9be9f21060e4b5a44cf8003ae0ff: Status 404 returned error can't find the container with id f702cec9255f1199582b8f8f5734df8b0d0d9be9f21060e4b5a44cf8003ae0ff
Nov 01 06:27:13.024848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:13.024817    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:27:13.024931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:13.024885    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:27:13.024931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:13.024916    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:27:13.076450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:13.076425    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" event={"ID":"7e777d1c-f8aa-43e3-9263-670757d9239f","Type":"ContainerStarted","Data":"f702cec9255f1199582b8f8f5734df8b0d0d9be9f21060e4b5a44cf8003ae0ff"}
Nov 01 06:27:14.085093 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:14.085057    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" event={"ID":"7e777d1c-f8aa-43e3-9263-670757d9239f","Type":"ContainerStarted","Data":"19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319"}
Nov 01 06:27:14.596394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:14.596365    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:14.598097 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:14.598072    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:27:14.598227 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:14.598114    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 06:27:15.596513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:15.596449    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:27:15.596998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:15.596531    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 06:27:16.596639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:16.596608    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:27:16.597034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:16.596664    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 06:27:17.596815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:17.596775    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:27:17.597231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:17.596853    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 06:27:18.596344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:18.596309    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 01 06:27:18.596556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:18.596367    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 01 06:27:19.758838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:19.758809    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:19.759506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:19.759318    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:19.760279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:19.760260    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:27:19.977542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:19.977455    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podStartSLOduration=7.977436859 podStartE2EDuration="7.977436859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:14.299214517 +0000 UTC m=+1045.867021347" watchObservedRunningTime="2026-11-01 06:27:19.977436859 +0000 UTC m=+1051.545243690"
Nov 01 06:27:22.615931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:22.615899    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:27:24.243724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.243683    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" podUID="159db47b-55fb-423a-821b-7567be5cd07d" containerName="main" containerID="cri-o://e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c" gracePeriod=45
Nov 01 06:27:24.803732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.803699    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:27:24.897070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.896978    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm\") pod \"159db47b-55fb-423a-821b-7567be5cd07d\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") "
Nov 01 06:27:24.897242 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897110    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs\" (UniqueName: \"kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs\") pod \"159db47b-55fb-423a-821b-7567be5cd07d\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") "
Nov 01 06:27:24.897242 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897142    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir\") pod \"159db47b-55fb-423a-821b-7567be5cd07d\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") "
Nov 01 06:27:24.897242 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897199    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache\") pod \"159db47b-55fb-423a-821b-7567be5cd07d\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") "
Nov 01 06:27:24.897407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897258    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm\" (UniqueName: \"kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm\") pod \"159db47b-55fb-423a-821b-7567be5cd07d\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") "
Nov 01 06:27:24.897407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897332    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home\") pod \"159db47b-55fb-423a-821b-7567be5cd07d\" (UID: \"159db47b-55fb-423a-821b-7567be5cd07d\") "
Nov 01 06:27:24.897531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897434    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache" pod "159db47b-55fb-423a-821b-7567be5cd07d" (UID: "159db47b-55fb-423a-821b-7567be5cd07d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:27:24.897531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897507    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir" pod "159db47b-55fb-423a-821b-7567be5cd07d" (UID: "159db47b-55fb-423a-821b-7567be5cd07d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:27:24.897740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.897720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home" pod "159db47b-55fb-423a-821b-7567be5cd07d" (UID: "159db47b-55fb-423a-821b-7567be5cd07d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:27:24.898142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.898085    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:24.898142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.898110    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:24.898142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.898125    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:24.899825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.899768    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm" pod "159db47b-55fb-423a-821b-7567be5cd07d" (UID: "159db47b-55fb-423a-821b-7567be5cd07d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:27:24.899941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.899835    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs" pod "159db47b-55fb-423a-821b-7567be5cd07d" (UID: "159db47b-55fb-423a-821b-7567be5cd07d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:27:24.902305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.902269    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm" pod "159db47b-55fb-423a-821b-7567be5cd07d" (UID: "159db47b-55fb-423a-821b-7567be5cd07d"). InnerVolumeSpecName "kube-api-access-t4mtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:27:24.999152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.999117    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159db47b-55fb-423a-821b-7567be5cd07d-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:24.999152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.999146    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159db47b-55fb-423a-821b-7567be5cd07d-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:24.999152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:24.999157    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mtm\" (UniqueName: \"kubernetes.io/projected/159db47b-55fb-423a-821b-7567be5cd07d-kube-api-access-t4mtm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:25.249400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.249320    4468 generic.go:356] "Generic (PLEG): container finished" podID="159db47b-55fb-423a-821b-7567be5cd07d" containerID="e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c" exitCode=0
Nov 01 06:27:25.249400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.249358    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"
Nov 01 06:27:25.249400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.249368    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" event={"ID":"159db47b-55fb-423a-821b-7567be5cd07d","Type":"ContainerDied","Data":"e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c"}
Nov 01 06:27:25.250024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.249404    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" event={"ID":"159db47b-55fb-423a-821b-7567be5cd07d","Type":"ContainerDied","Data":"803c63e294ba7e26e4793137908c53ef5596c67aaa018bbb1492aff34d058b62"}
Nov 01 06:27:25.250024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.249422    4468 scope.go:122] "RemoveContainer" containerID="e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c"
Nov 01 06:27:25.262773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.262752    4468 scope.go:122] "RemoveContainer" containerID="e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c"
Nov 01 06:27:25.263045 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:27:25.263023    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c\": container with ID starting with e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c not found: ID does not exist" containerID="e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c"
Nov 01 06:27:25.263106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.263056    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c"} err="failed to get container status \"e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c\": rpc error: code = NotFound desc = could not find container \"e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c\": container with ID starting with e15c61ddd33accc3d56a50a5fed362b327280b41317e39ea3650bbd312ec7e3c not found: ID does not exist"
Nov 01 06:27:25.413042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.413008    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"]
Nov 01 06:27:25.416857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.416834    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8"]
Nov 01 06:27:25.712347 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:25.712288    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ef9b55/test-model-ef9b55-kserve-77fb678b7-pjrv8" podUID="159db47b-55fb-423a-821b-7567be5cd07d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:27:26.590140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:26.590107    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159db47b-55fb-423a-821b-7567be5cd07d" path="/var/lib/kubelet/pods/159db47b-55fb-423a-821b-7567be5cd07d/volumes"
Nov 01 06:27:32.947501 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:27:32.947448    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:27:37.128075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.128032    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"]
Nov 01 06:27:37.129411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.129382    4468 cpu_manager.go:414] "containerMap: removing container" podUID="159db47b-55fb-423a-821b-7567be5cd07d" containerName="main"
Nov 01 06:27:37.129411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.129411    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159db47b-55fb-423a-821b-7567be5cd07d" containerName="main"
Nov 01 06:27:37.129806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.129788    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="159db47b-55fb-423a-821b-7567be5cd07d" containerName="main"
Nov 01 06:27:37.134724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.134693    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.187125 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.187093    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"]
Nov 01 06:27:37.232185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.232150    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:27:37.232185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.232148    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:27:37.232185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.232162    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:27:37.232509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.232210    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:27:37.290623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.290578    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.291054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.290998    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.291232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.291145    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mzw8\" (UniqueName: \"kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.392983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.392898    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.393149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.393036    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.393149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.393123    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mzw8\" (UniqueName: \"kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.393820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.393799    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.395526 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.395501    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.449634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.449598    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mzw8\" (UniqueName: \"kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8\") pod \"maas-api-revoke-sub-95eecc25-npsfs\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:37.749410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:37.749319    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:38.128660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:38.128631    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"]
Nov 01 06:27:38.129103 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:27:38.129080    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f62809_f845_43c3_84b6_5bf1bfe7aac1.slice/crio-4dd95cf8b76128b8127c89d876474d8e0c8421747a4602312ce43226b08a99ed WatchSource:0}: Error finding container 4dd95cf8b76128b8127c89d876474d8e0c8421747a4602312ce43226b08a99ed: Status 404 returned error can't find the container with id 4dd95cf8b76128b8127c89d876474d8e0c8421747a4602312ce43226b08a99ed
Nov 01 06:27:38.338102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:38.338067    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs" event={"ID":"35f62809-f845-43c3-84b6-5bf1bfe7aac1","Type":"ContainerStarted","Data":"4dd95cf8b76128b8127c89d876474d8e0c8421747a4602312ce43226b08a99ed"}
Nov 01 06:27:39.346074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:39.346037    4468 generic.go:356] "Generic (PLEG): container finished" podID="35f62809-f845-43c3-84b6-5bf1bfe7aac1" containerID="cad0c361c9a745500d39d1b9df4768dab00e5fe14cb1d364caba0614913cd7b8" exitCode=0
Nov 01 06:27:39.346556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:39.346078    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs" event={"ID":"35f62809-f845-43c3-84b6-5bf1bfe7aac1","Type":"ContainerDied","Data":"cad0c361c9a745500d39d1b9df4768dab00e5fe14cb1d364caba0614913cd7b8"}
Nov 01 06:27:40.524222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.524196    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:40.633957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.633861    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca\") pod \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") "
Nov 01 06:27:40.634144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.633993    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca\") pod \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") "
Nov 01 06:27:40.634144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.634035    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8\" (UniqueName: \"kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8\") pod \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\" (UID: \"35f62809-f845-43c3-84b6-5bf1bfe7aac1\") "
Nov 01 06:27:40.634431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.634402    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca" pod "35f62809-f845-43c3-84b6-5bf1bfe7aac1" (UID: "35f62809-f845-43c3-84b6-5bf1bfe7aac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:27:40.634562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.634539    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:40.636735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.636701    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca" pod "35f62809-f845-43c3-84b6-5bf1bfe7aac1" (UID: "35f62809-f845-43c3-84b6-5bf1bfe7aac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:27:40.636844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.636783    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8" pod "35f62809-f845-43c3-84b6-5bf1bfe7aac1" (UID: "35f62809-f845-43c3-84b6-5bf1bfe7aac1"). InnerVolumeSpecName "kube-api-access-7mzw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:27:40.735775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.735722    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mzw8\" (UniqueName: \"kubernetes.io/projected/35f62809-f845-43c3-84b6-5bf1bfe7aac1-kube-api-access-7mzw8\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:40.735775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:40.735762    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f62809-f845-43c3-84b6-5bf1bfe7aac1-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:41.360444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:41.360408    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs" event={"ID":"35f62809-f845-43c3-84b6-5bf1bfe7aac1","Type":"ContainerDied","Data":"4dd95cf8b76128b8127c89d876474d8e0c8421747a4602312ce43226b08a99ed"}
Nov 01 06:27:41.360633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:41.360500    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dd95cf8b76128b8127c89d876474d8e0c8421747a4602312ce43226b08a99ed"
Nov 01 06:27:41.360633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:41.360418    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"
Nov 01 06:27:43.036267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:43.036216    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:27:49.552142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:49.552111    4468 scope.go:122] "RemoveContainer" containerID="9a82f489fb6e61fb93205e01844f27bb98dd9674a1b100a26f248013dbf7808a"
Nov 01 06:27:52.617432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.617391    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"]
Nov 01 06:27:52.617963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.617948    4468 cpu_manager.go:414] "containerMap: removing container" podUID="35f62809-f845-43c3-84b6-5bf1bfe7aac1" containerName="revoke-keys"
Nov 01 06:27:52.618010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.617964    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f62809-f845-43c3-84b6-5bf1bfe7aac1" containerName="revoke-keys"
Nov 01 06:27:52.618080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.618071    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f62809-f845-43c3-84b6-5bf1bfe7aac1" containerName="revoke-keys"
Nov 01 06:27:52.623111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.623091    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.677777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.677746    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"]
Nov 01 06:27:52.726923 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.726890    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:27:52.726923 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.726890    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:27:52.726923 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.726897    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:27:52.728422 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.728399    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:27:52.740367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.740333    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.740529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.740372    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9pd\" (UniqueName: \"kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.740602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.740580    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.841650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.841610    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9pd\" (UniqueName: \"kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.841830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.841734    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.841830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.841796    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.842412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.842377    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.844252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.844234    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.899312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.899242    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9pd\" (UniqueName: \"kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd\") pod \"maas-api-revoke-sub-818428b6-x2zng\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:52.936843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:52.936815    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:53.310087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:53.310045    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"]
Nov 01 06:27:53.310583 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:27:53.310546    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb388dc36_3146_41bc_9ba9_59023025de3c.slice/crio-554ae068b78ad1c5437301c216ab46aa8c1028e8bda4b87cec78f1007ebafffd WatchSource:0}: Error finding container 554ae068b78ad1c5437301c216ab46aa8c1028e8bda4b87cec78f1007ebafffd: Status 404 returned error can't find the container with id 554ae068b78ad1c5437301c216ab46aa8c1028e8bda4b87cec78f1007ebafffd
Nov 01 06:27:53.474587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:53.474536    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng" event={"ID":"b388dc36-3146-41bc-9ba9-59023025de3c","Type":"ContainerStarted","Data":"554ae068b78ad1c5437301c216ab46aa8c1028e8bda4b87cec78f1007ebafffd"}
Nov 01 06:27:54.489690 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:54.489584    4468 generic.go:356] "Generic (PLEG): container finished" podID="b388dc36-3146-41bc-9ba9-59023025de3c" containerID="6d2fbb864e8c57e3cf0fdcbf8d2e3843c0adc151421a8afc2c4b53a7977ffeb3" exitCode=0
Nov 01 06:27:54.490342 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:54.489689    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng" event={"ID":"b388dc36-3146-41bc-9ba9-59023025de3c","Type":"ContainerDied","Data":"6d2fbb864e8c57e3cf0fdcbf8d2e3843c0adc151421a8afc2c4b53a7977ffeb3"}
Nov 01 06:27:54.804000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:54.803963    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"]
Nov 01 06:27:54.804235 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:54.804209    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" containerID="cri-o://27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91" gracePeriod=30
Nov 01 06:27:55.381662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.381625    4468 patch_prober.go:29] interesting pod/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:27:55.381868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.381697    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:27:55.655372 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.655342    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:55.681336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.681302    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca\") pod \"b388dc36-3146-41bc-9ba9-59023025de3c\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") "
Nov 01 06:27:55.681543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.681425    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd\" (UniqueName: \"kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd\") pod \"b388dc36-3146-41bc-9ba9-59023025de3c\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") "
Nov 01 06:27:55.681689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.681666    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca" pod "b388dc36-3146-41bc-9ba9-59023025de3c" (UID: "b388dc36-3146-41bc-9ba9-59023025de3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:27:55.681852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.681825    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:55.683743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.683720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd" pod "b388dc36-3146-41bc-9ba9-59023025de3c" (UID: "b388dc36-3146-41bc-9ba9-59023025de3c"). InnerVolumeSpecName "kube-api-access-9v9pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:27:55.782520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.782484    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca\") pod \"b388dc36-3146-41bc-9ba9-59023025de3c\" (UID: \"b388dc36-3146-41bc-9ba9-59023025de3c\") "
Nov 01 06:27:55.783126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.783102    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9pd\" (UniqueName: \"kubernetes.io/projected/b388dc36-3146-41bc-9ba9-59023025de3c-kube-api-access-9v9pd\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:55.784808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.784782    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca" pod "b388dc36-3146-41bc-9ba9-59023025de3c" (UID: "b388dc36-3146-41bc-9ba9-59023025de3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:27:55.884896 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:55.884867    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b388dc36-3146-41bc-9ba9-59023025de3c-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:56.507972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.507937    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"
Nov 01 06:27:56.507972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.507967    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng" event={"ID":"b388dc36-3146-41bc-9ba9-59023025de3c","Type":"ContainerDied","Data":"554ae068b78ad1c5437301c216ab46aa8c1028e8bda4b87cec78f1007ebafffd"}
Nov 01 06:27:56.508193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.508018    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="554ae068b78ad1c5437301c216ab46aa8c1028e8bda4b87cec78f1007ebafffd"
Nov 01 06:27:56.863259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.863226    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"]
Nov 01 06:27:56.864031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.864010    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b388dc36-3146-41bc-9ba9-59023025de3c" containerName="revoke-keys"
Nov 01 06:27:56.864129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.864031    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b388dc36-3146-41bc-9ba9-59023025de3c" containerName="revoke-keys"
Nov 01 06:27:56.864258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.864243    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b388dc36-3146-41bc-9ba9-59023025de3c" containerName="revoke-keys"
Nov 01 06:27:56.868680 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.868659    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:56.922919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.922882    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"]
Nov 01 06:27:56.944288 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.944249    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7kk\" (UniqueName: \"kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:56.944624 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.944594    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:56.965814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.965780    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd9d24\""
Nov 01 06:27:56.966229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:56.966192    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd9d24-dockercfg-hvvt5\""
Nov 01 06:27:57.048289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.048243    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7kk\" (UniqueName: \"kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:57.061735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.061697    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:57.063646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.063606    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:57.126006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.125915    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7kk\" (UniqueName: \"kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:57.183859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.183819    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:27:57.650498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.646077    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"]
Nov 01 06:27:57.667654 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.667621    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"]
Nov 01 06:27:57.696494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.692990    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:57.723490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.722051    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"]
Nov 01 06:27:57.872504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.843350    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpc28\" (UniqueName: \"kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28\") pod \"payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:57.899345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.899313    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:58.050504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:57.906803    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.163728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.163702    4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:27:58.219563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.219521    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.219898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.219874    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.220061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220042    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.220189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220173    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.220324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220306    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.220458 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220440    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88sx\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.220660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220639    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpc28\" (UniqueName: \"kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28\") pod \"payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:58.220824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220808    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.221001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.220984    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:58.221140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.221123    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.221314 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.221297    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.223020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.222995    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:58.228016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.227586    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"]
Nov 01 06:27:58.260710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.260601    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"]
Nov 01 06:27:58.300312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.299163    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpc28\" (UniqueName: \"kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28\") pod \"payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:58.326060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.326019    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.326378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.326355    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.326650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.326630    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.326823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.326805    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.326951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.326935    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.327058 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.327045    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.327162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.327148    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.327280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.327264    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88sx\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.327452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.327435    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.331628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.331599    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.332360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.332334    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.333122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.333100    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.338503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.335094    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.338503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.336076    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.379703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.378646    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.393817 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.389620    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.415220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.400052    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.415220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.411623    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88sx\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.516105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.516002    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:27:58.633122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.633077    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:27:58.722963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.722887    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9" event={"ID":"a5d94d75-dde2-42b5-9aee-cfdba3875810","Type":"ContainerStarted","Data":"a6cf64b59f29f15d0fae969104450906b7e190cb963fafd6495d5fac6f090bf0"}
Nov 01 06:27:58.900409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:58.900377    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"]
Nov 01 06:27:58.901447 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:27:58.901421    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1817c8_1355_456e_a943_5f5d045df3c6.slice/crio-1a49618513a78677ba5bce4d18e524403a93c1f63f4e1f2cd4e96b9090d24f38 WatchSource:0}: Error finding container 1a49618513a78677ba5bce4d18e524403a93c1f63f4e1f2cd4e96b9090d24f38: Status 404 returned error can't find the container with id 1a49618513a78677ba5bce4d18e524403a93c1f63f4e1f2cd4e96b9090d24f38
Nov 01 06:27:59.013413 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:27:59.013382    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7274ce5d_f935_4730_94ef_1fda35773de2.slice/crio-aff4411082bf9c143b4fa5a9ae70a55fb60f9173ab023c45bd5877cdad5da685 WatchSource:0}: Error finding container aff4411082bf9c143b4fa5a9ae70a55fb60f9173ab023c45bd5877cdad5da685: Status 404 returned error can't find the container with id aff4411082bf9c143b4fa5a9ae70a55fb60f9173ab023c45bd5877cdad5da685
Nov 01 06:27:59.015368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.015339    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"]
Nov 01 06:27:59.016048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.016015    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:27:59.016117 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.016105    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:27:59.016152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.016137    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:27:59.737003 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.736904    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl" event={"ID":"ca1817c8-1355-456e-a943-5f5d045df3c6","Type":"ContainerStarted","Data":"1a49618513a78677ba5bce4d18e524403a93c1f63f4e1f2cd4e96b9090d24f38"}
Nov 01 06:27:59.740168 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.740055    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" event={"ID":"7274ce5d-f935-4730-94ef-1fda35773de2","Type":"ContainerStarted","Data":"36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80"}
Nov 01 06:27:59.740522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.740491    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" event={"ID":"7274ce5d-f935-4730-94ef-1fda35773de2","Type":"ContainerStarted","Data":"aff4411082bf9c143b4fa5a9ae70a55fb60f9173ab023c45bd5877cdad5da685"}
Nov 01 06:27:59.908702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:27:59.908635    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podStartSLOduration=2.90861293 podStartE2EDuration="2.90861293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:59.903947875 +0000 UTC m=+1091.471754708" watchObservedRunningTime="2026-11-01 06:27:59.90861293 +0000 UTC m=+1091.476419759"
Nov 01 06:28:00.057367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.057288    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"]
Nov 01 06:28:00.065716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.063600    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca54dda5-6csmc"]
Nov 01 06:28:00.200692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.200660    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.366689    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.366756    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.366803    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.366827    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.366917    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.366948    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.367065    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.367145    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.370640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.367208    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy\") pod \"b4c31f27-e3c6-40e5-895c-9cae71d50475\" (UID: \"b4c31f27-e3c6-40e5-895c-9cae71d50475\") "
Nov 01 06:28:00.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.372056    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:28:00.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.372160    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:28:00.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.372242    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:28:00.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.372296    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:28:00.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.372809    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:00.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.373137    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "kube-api-access-vb6lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:00.375516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.374278    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.375516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.374307    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c31f27-e3c6-40e5-895c-9cae71d50475-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.375516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.374323    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.375516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.374337    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.375516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.374351    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6lp\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-kube-api-access-vb6lp\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.375516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.374365    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.378062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.377914    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:00.378174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.378111    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:28:00.379145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.379094    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy" pod "b4c31f27-e3c6-40e5-895c-9cae71d50475" (UID: "b4c31f27-e3c6-40e5-895c-9cae71d50475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:28:00.476129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.476092    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.476129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.476132    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.476377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.476147    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c31f27-e3c6-40e5-895c-9cae71d50475-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:00.594928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.594889    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9" path="/var/lib/kubelet/pods/cd0a8b0d-b7a8-4af8-b3e7-3c1cccc9c3f9/volumes"
Nov 01 06:28:00.633614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.633530    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:28:00.635314 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.635282    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 06:28:00.635490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.635340    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 06:28:00.750673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.750636    4468 generic.go:356] "Generic (PLEG): container finished" podID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerID="27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91" exitCode=0
Nov 01 06:28:00.751378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.751362    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"
Nov 01 06:28:00.751802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.751785    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" event={"ID":"b4c31f27-e3c6-40e5-895c-9cae71d50475","Type":"ContainerDied","Data":"27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91"}
Nov 01 06:28:00.751886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.751823    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s" event={"ID":"b4c31f27-e3c6-40e5-895c-9cae71d50475","Type":"ContainerDied","Data":"313c541c920dac1e74325e598fd0f7da73465f69e588688c7fcac98e8b122374"}
Nov 01 06:28:00.751886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.751843    4468 scope.go:122] "RemoveContainer" containerID="27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91"
Nov 01 06:28:00.768685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.768575    4468 scope.go:122] "RemoveContainer" containerID="85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"
Nov 01 06:28:00.781725 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.781679    4468 scope.go:122] "RemoveContainer" containerID="27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91"
Nov 01 06:28:00.782244 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:28:00.782096    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91\": container with ID starting with 27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91 not found: ID does not exist" containerID="27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91"
Nov 01 06:28:00.782244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.782136    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91"} err="failed to get container status \"27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91\": rpc error: code = NotFound desc = could not find container \"27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91\": container with ID starting with 27f40335600c064bde166e0fe327ade016dc90aa577ffffcf2b429672dc41f91 not found: ID does not exist"
Nov 01 06:28:00.782244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.782160    4468 scope.go:122] "RemoveContainer" containerID="85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"
Nov 01 06:28:00.782532 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:28:00.782508    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549\": container with ID starting with 85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549 not found: ID does not exist" containerID="85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"
Nov 01 06:28:00.782608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.782541    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549"} err="failed to get container status \"85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549\": rpc error: code = NotFound desc = could not find container \"85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549\": container with ID starting with 85e1a47880a08e54b47d9558ac35431ed512889cf826f713f6ff74114ef78549 not found: ID does not exist"
Nov 01 06:28:00.945777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.945654    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"]
Nov 01 06:28:00.950860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:00.950829    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-85584b-openshift-default-879d4df9f-6qm5s"]
Nov 01 06:28:01.634310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:01.634273    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 06:28:01.634547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:01.634335    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 06:28:01.759806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:01.759770    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9" event={"ID":"a5d94d75-dde2-42b5-9aee-cfdba3875810","Type":"ContainerStarted","Data":"76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578"}
Nov 01 06:28:01.759993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:01.759979    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:28:01.877376 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:01.877318    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9" podStartSLOduration=2.669551324 podStartE2EDuration="5.877300489s" totalImagesPullingTime="3.207749165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:01.873733835 +0000 UTC m=+1093.441540666" watchObservedRunningTime="2026-11-01 06:28:01.877300489 +0000 UTC m=+1093.445107317"
Nov 01 06:28:02.598137 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:02.598104    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" path="/var/lib/kubelet/pods/b4c31f27-e3c6-40e5-895c-9cae71d50475/volumes"
Nov 01 06:28:02.646604 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:02.644590    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 06:28:02.646604 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:02.644654    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 06:28:02.774419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:02.774356    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl" event={"ID":"ca1817c8-1355-456e-a943-5f5d045df3c6","Type":"ContainerStarted","Data":"b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330"}
Nov 01 06:28:02.774770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:02.774748    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:28:02.897019 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:02.896900    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl" podStartSLOduration=3.10436195 podStartE2EDuration="5.896883424s" totalImagesPullingTime="2.792521474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:02.894203262 +0000 UTC m=+1094.462010097" watchObservedRunningTime="2026-11-01 06:28:02.896883424 +0000 UTC m=+1094.464690255"
Nov 01 06:28:02.980656 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:28:02.980617    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:28:03.634449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:03.634411    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 06:28:03.634897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:03.634501    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 06:28:04.634192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:04.634089    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 06:28:04.634192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:04.634147    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 06:28:05.633943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:05.633901    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 06:28:05.634359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:05.633972    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 06:28:06.750062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:06.750021    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:28:06.750665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:06.750515    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:28:06.751861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:06.751837    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:28:09.041971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.041932    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"]
Nov 01 06:28:09.042538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.042520    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy"
Nov 01 06:28:09.042601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.042539    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy"
Nov 01 06:28:09.042683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.042672    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy"
Nov 01 06:28:09.042737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.042686    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy"
Nov 01 06:28:09.046190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.046172    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.100880 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.100837    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:28:09.131064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.131015    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.131274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.131230    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.131360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.131297    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvg77\" (UniqueName: \"kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.150569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.150446    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:28:09.150569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.150544    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:28:09.150569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.150544    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:28:09.154453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.154421    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"]
Nov 01 06:28:09.232868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.232824    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.233048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.232935    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.233048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.232969    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvg77\" (UniqueName: \"kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.233663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.233639    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.235432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.235413    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.290042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.289996    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvg77\" (UniqueName: \"kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77\") pod \"maas-api-revoke-sub-45361c63-ljt6b\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.359676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.359632    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:09.743365 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.743298    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"]
Nov 01 06:28:09.743563 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:28:09.743537    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce6df7b_e3ac_484e_8b71_8f72a11318e5.slice/crio-57eaa717df0fb3a2f389500a38ae6c92d9f574be8861b69ff8c74748c26457ca WatchSource:0}: Error finding container 57eaa717df0fb3a2f389500a38ae6c92d9f574be8861b69ff8c74748c26457ca: Status 404 returned error can't find the container with id 57eaa717df0fb3a2f389500a38ae6c92d9f574be8861b69ff8c74748c26457ca
Nov 01 06:28:09.893514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:09.893449    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b" event={"ID":"3ce6df7b-e3ac-484e-8b71-8f72a11318e5","Type":"ContainerStarted","Data":"57eaa717df0fb3a2f389500a38ae6c92d9f574be8861b69ff8c74748c26457ca"}
Nov 01 06:28:10.902403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:10.902308    4468 generic.go:356] "Generic (PLEG): container finished" podID="3ce6df7b-e3ac-484e-8b71-8f72a11318e5" containerID="2048b098b966ba9a5d65c0946cd438bb48b4ba3201abecaef77efe30adc31ab0" exitCode=0
Nov 01 06:28:10.902852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:10.902396    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b" event={"ID":"3ce6df7b-e3ac-484e-8b71-8f72a11318e5","Type":"ContainerDied","Data":"2048b098b966ba9a5d65c0946cd438bb48b4ba3201abecaef77efe30adc31ab0"}
Nov 01 06:28:12.062733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.062707    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:12.169692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.169591    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:28:12.170146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.170111    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca\") pod \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") "
Nov 01 06:28:12.170251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.170206    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca\") pod \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") "
Nov 01 06:28:12.170378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.170362    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77\" (UniqueName: \"kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77\") pod \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\" (UID: \"3ce6df7b-e3ac-484e-8b71-8f72a11318e5\") "
Nov 01 06:28:12.172020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.171557    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca" pod "3ce6df7b-e3ac-484e-8b71-8f72a11318e5" (UID: "3ce6df7b-e3ac-484e-8b71-8f72a11318e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:12.189889 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.188105    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca" pod "3ce6df7b-e3ac-484e-8b71-8f72a11318e5" (UID: "3ce6df7b-e3ac-484e-8b71-8f72a11318e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:28:12.190958 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.190925    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77" pod "3ce6df7b-e3ac-484e-8b71-8f72a11318e5" (UID: "3ce6df7b-e3ac-484e-8b71-8f72a11318e5"). InnerVolumeSpecName "kube-api-access-xvg77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:12.272031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.271994    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:12.272208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.272112    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:12.272249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.272231    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvg77\" (UniqueName: \"kubernetes.io/projected/3ce6df7b-e3ac-484e-8b71-8f72a11318e5-kube-api-access-xvg77\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:12.817922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.816537    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:28:12.920260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.920162    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"
Nov 01 06:28:12.920260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.920193    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b" event={"ID":"3ce6df7b-e3ac-484e-8b71-8f72a11318e5","Type":"ContainerDied","Data":"57eaa717df0fb3a2f389500a38ae6c92d9f574be8861b69ff8c74748c26457ca"}
Nov 01 06:28:12.920260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:12.920243    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57eaa717df0fb3a2f389500a38ae6c92d9f574be8861b69ff8c74748c26457ca"
Nov 01 06:28:13.981499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:13.979975    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:28:15.288494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:15.270630    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx" podUID="82915854-11e3-466d-a38e-2f235096f61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:28:15.371497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:15.370598    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 01 06:28:19.045118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:19.045084    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"]
Nov 01 06:28:19.104206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:19.104151    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e11976-qq9js"]
Nov 01 06:28:20.590806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:20.590773    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11bc217b-862d-428a-8bc0-3991f32a288e" path="/var/lib/kubelet/pods/11bc217b-862d-428a-8bc0-3991f32a288e/volumes"
Nov 01 06:28:25.782976 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.782944    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"]
Nov 01 06:28:25.783505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.783478    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ce6df7b-e3ac-484e-8b71-8f72a11318e5" containerName="revoke-keys"
Nov 01 06:28:25.783505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.783491    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce6df7b-e3ac-484e-8b71-8f72a11318e5" containerName="revoke-keys"
Nov 01 06:28:25.783580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.783537    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy"
Nov 01 06:28:25.783580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.783542    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c31f27-e3c6-40e5-895c-9cae71d50475" containerName="istio-proxy"
Nov 01 06:28:25.783651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.783622    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce6df7b-e3ac-484e-8b71-8f72a11318e5" containerName="revoke-keys"
Nov 01 06:28:25.788242 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.788224    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.859069 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.859034    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e7b2a3-openshift-default-dockercfg-7q695\""
Nov 01 06:28:25.895997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.895962    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"]
Nov 01 06:28:25.931802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.931760    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.931992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.931838    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.931992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.931929    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.932092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.931986    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.932092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.932028    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.932092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.932056    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.932260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.932233    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7645x\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.932313 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.932291    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:25.932449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:25.932432    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038390    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038454    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038543    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038589    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038650    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038701    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038744    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038903    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7645x\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.038968    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.039242    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.039907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.039380    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.040946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.040161    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.041006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.040972    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.041204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.041179    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.045457 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.045425    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.053868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.053796    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.096016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.095984    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7645x\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.096211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.096194    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token\") pod \"e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") " pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.102213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.102191    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:26.559502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.559438    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"]
Nov 01 06:28:26.559872 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:28:26.559851    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48a655c_a6fe_4e33_b42f_e79b37b7c142.slice/crio-bd85c2475aa64a59eaa84ff7c747fd95aa9e02b0369d361d94619fa7f1f52ede WatchSource:0}: Error finding container bd85c2475aa64a59eaa84ff7c747fd95aa9e02b0369d361d94619fa7f1f52ede: Status 404 returned error can't find the container with id bd85c2475aa64a59eaa84ff7c747fd95aa9e02b0369d361d94619fa7f1f52ede
Nov 01 06:28:26.562167 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.562129    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:28:26.562275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.562209    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:28:26.562275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.562252    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:28:26.704651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.704615    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" event={"ID":"b48a655c-a6fe-4e33-b42f-e79b37b7c142","Type":"ContainerStarted","Data":"dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"}
Nov 01 06:28:26.704823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.704664    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" event={"ID":"b48a655c-a6fe-4e33-b42f-e79b37b7c142","Type":"ContainerStarted","Data":"bd85c2475aa64a59eaa84ff7c747fd95aa9e02b0369d361d94619fa7f1f52ede"}
Nov 01 06:28:26.826511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:26.826322    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podStartSLOduration=1.826288995 podStartE2EDuration="1.826288995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:26.82445059 +0000 UTC m=+1118.392257423" watchObservedRunningTime="2026-11-01 06:28:26.826288995 +0000 UTC m=+1118.394095823"
Nov 01 06:28:27.103235 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:27.103148    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:27.109359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:27.109335    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:27.711984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:27.711950    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:27.713129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:27.713110    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:28:32.952256 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:28:32.952161    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:28:37.044858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:37.044824    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"]
Nov 01 06:28:37.094238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:37.094201    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1de3d-bww8z"]
Nov 01 06:28:38.591988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:38.591951    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dd073e-2167-4589-ba0c-3249614fe832" path="/var/lib/kubelet/pods/f2dd073e-2167-4589-ba0c-3249614fe832/volumes"
Nov 01 06:28:47.207619 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:47.207586    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:28:48.016577 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.016541    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"]
Nov 01 06:28:48.022967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.022938    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.066829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.066792    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"]
Nov 01 06:28:48.111143 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.111077    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:28:48.111404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.111127    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:28:48.111404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.111171    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:28:48.125800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.125778    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:28:48.173454 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.173413    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlg5n\" (UniqueName: \"kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.173647 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.173512    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.173813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.173779    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.282552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.282317    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.283079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.282802    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.283079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.282983    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg5n\" (UniqueName: \"kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.301599 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.294295    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.331991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.331952    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.349127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.347686    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlg5n\" (UniqueName: \"kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n\") pod \"maas-api-revoke-sub-d010850d-cjbgk\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:48.651035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:48.650764    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:49.156545 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.156511    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cq9k6-jlkgw"]
Nov 01 06:28:49.608831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.608793    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cq9k6-jlkgw"]
Nov 01 06:28:49.610557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.609670    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.704603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.693063    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 06:28:49.725258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.725173    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.725655 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.725607    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.725869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.725835    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.726563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.726232    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kq6j\" (UniqueName: \"kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.854213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.831283    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.854213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.831384    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.854213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.831427    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.854213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.831509    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kq6j\" (UniqueName: \"kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.854213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.832803    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.854213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.833087    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:49.923491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.920275    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 06:28:49.923491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.920424    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jp6zc\""
Nov 01 06:28:49.943667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:49.943599    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:50.204487 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:50.196821    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kq6j\" (UniqueName: \"kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j\") pod \"periodic-gathering-cq9k6-jlkgw\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") " pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:50.243492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:50.243225    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:28:50.323492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:50.323149    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:28:50.334929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:50.334896    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"]
Nov 01 06:28:51.078121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:51.078086    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cq9k6-jlkgw"]
Nov 01 06:28:51.079552 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:28:51.079516    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388bf2ac_33b8_40c2_a4c1_085dc435a3fa.slice/crio-49d0e12e060b10bcabd30c873ca8466880aefacb1782aa3e493ce15c5e9174d0 WatchSource:0}: Error finding container 49d0e12e060b10bcabd30c873ca8466880aefacb1782aa3e493ce15c5e9174d0: Status 404 returned error can't find the container with id 49d0e12e060b10bcabd30c873ca8466880aefacb1782aa3e493ce15c5e9174d0
Nov 01 06:28:51.111539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:51.111502    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw" event={"ID":"388bf2ac-33b8-40c2-a4c1-085dc435a3fa","Type":"ContainerStarted","Data":"49d0e12e060b10bcabd30c873ca8466880aefacb1782aa3e493ce15c5e9174d0"}
Nov 01 06:28:51.113066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:51.113037    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk" event={"ID":"27f499dd-81b4-4cef-98c5-81e15cdf8c29","Type":"ContainerStarted","Data":"3b0d41306208e63d6665570e247002bcf11eae933b6f3728be28d47109bde5fe"}
Nov 01 06:28:51.113177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:51.113088    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk" event={"ID":"27f499dd-81b4-4cef-98c5-81e15cdf8c29","Type":"ContainerStarted","Data":"3fbd5d2359c45a2d4ba8fedc4bd1cd97e8b04d04bf070ade69dfccde2af65a61"}
Nov 01 06:28:51.230206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:51.230084    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk" podStartSLOduration=4.230070905 podStartE2EDuration="4.230070905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:51.226983084 +0000 UTC m=+1142.794789923" watchObservedRunningTime="2026-11-01 06:28:51.230070905 +0000 UTC m=+1142.797877734"
Nov 01 06:28:52.123158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:52.123120    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw" event={"ID":"388bf2ac-33b8-40c2-a4c1-085dc435a3fa","Type":"ContainerStarted","Data":"16eee90fe72de21487b0acf66e4abf8f36035ffc1ed64b0e479fc13644e3b320"}
Nov 01 06:28:52.125359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:52.125328    4468 generic.go:356] "Generic (PLEG): container finished" podID="27f499dd-81b4-4cef-98c5-81e15cdf8c29" containerID="3b0d41306208e63d6665570e247002bcf11eae933b6f3728be28d47109bde5fe" exitCode=0
Nov 01 06:28:52.125558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:52.125539    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk" event={"ID":"27f499dd-81b4-4cef-98c5-81e15cdf8c29","Type":"ContainerDied","Data":"3b0d41306208e63d6665570e247002bcf11eae933b6f3728be28d47109bde5fe"}
Nov 01 06:28:52.240096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:52.240041    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw" podStartSLOduration=3.240025179 podStartE2EDuration="3.240025179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:52.237258444 +0000 UTC m=+1143.805065301" watchObservedRunningTime="2026-11-01 06:28:52.240025179 +0000 UTC m=+1143.807832042"
Nov 01 06:28:53.274449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.274419    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:53.316132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.316101    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca\") pod \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") "
Nov 01 06:28:53.316338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.316160    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n\" (UniqueName: \"kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n\") pod \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") "
Nov 01 06:28:53.316418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.316347    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca\") pod \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\" (UID: \"27f499dd-81b4-4cef-98c5-81e15cdf8c29\") "
Nov 01 06:28:53.316744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.316717    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca" pod "27f499dd-81b4-4cef-98c5-81e15cdf8c29" (UID: "27f499dd-81b4-4cef-98c5-81e15cdf8c29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:53.316882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.316865    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:53.318447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.318421    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca" pod "27f499dd-81b4-4cef-98c5-81e15cdf8c29" (UID: "27f499dd-81b4-4cef-98c5-81e15cdf8c29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:28:53.318664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.318552    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n" pod "27f499dd-81b4-4cef-98c5-81e15cdf8c29" (UID: "27f499dd-81b4-4cef-98c5-81e15cdf8c29"). InnerVolumeSpecName "kube-api-access-mlg5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:53.419034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.418941    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f499dd-81b4-4cef-98c5-81e15cdf8c29-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:53.419034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:53.418987    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlg5n\" (UniqueName: \"kubernetes.io/projected/27f499dd-81b4-4cef-98c5-81e15cdf8c29-kube-api-access-mlg5n\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:54.140335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:54.140302    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"
Nov 01 06:28:54.140335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:54.140314    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk" event={"ID":"27f499dd-81b4-4cef-98c5-81e15cdf8c29","Type":"ContainerDied","Data":"3fbd5d2359c45a2d4ba8fedc4bd1cd97e8b04d04bf070ade69dfccde2af65a61"}
Nov 01 06:28:54.140569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:54.140361    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fbd5d2359c45a2d4ba8fedc4bd1cd97e8b04d04bf070ade69dfccde2af65a61"
Nov 01 06:28:55.608784 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.608750    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"]
Nov 01 06:28:55.609335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.609322    4468 cpu_manager.go:414] "containerMap: removing container" podUID="27f499dd-81b4-4cef-98c5-81e15cdf8c29" containerName="revoke-keys"
Nov 01 06:28:55.609382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.609336    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f499dd-81b4-4cef-98c5-81e15cdf8c29" containerName="revoke-keys"
Nov 01 06:28:55.609443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.609433    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f499dd-81b4-4cef-98c5-81e15cdf8c29" containerName="revoke-keys"
Nov 01 06:28:55.613101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.613080    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.646261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.646225    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.646429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.646392    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.646731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.646703    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnt2\" (UniqueName: \"kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.668998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.668960    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"]
Nov 01 06:28:55.713796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.713757    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:28:55.713796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.713795    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:28:55.714049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.713809    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:28:55.714049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.713839    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:28:55.747717 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.747674    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnt2\" (UniqueName: \"kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.747909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.747766    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.747909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.747834    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.748542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.748521    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.750297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.750276    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.804355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.804321    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnt2\" (UniqueName: \"kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2\") pod \"maas-api-revoke-sub-7ef364b2-tg562\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:55.927096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:55.927015    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:56.302989 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:56.302962    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"]
Nov 01 06:28:56.303566 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:28:56.303540    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0acf3f5_6233_422f_b186_96d1a3c53be7.slice/crio-cd98bd336c04c843cac0021a3c6ccb9ad0b8cc6832f422533426d8fb10ec55d4 WatchSource:0}: Error finding container cd98bd336c04c843cac0021a3c6ccb9ad0b8cc6832f422533426d8fb10ec55d4: Status 404 returned error can't find the container with id cd98bd336c04c843cac0021a3c6ccb9ad0b8cc6832f422533426d8fb10ec55d4
Nov 01 06:28:57.162106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:57.162074    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562" event={"ID":"b0acf3f5-6233-422f-b186-96d1a3c53be7","Type":"ContainerStarted","Data":"fb4c6872cab1de07b69b72c94c6ddb86d391fb9032441a437673e77cbca5d6a9"}
Nov 01 06:28:57.162644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:57.162127    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562" event={"ID":"b0acf3f5-6233-422f-b186-96d1a3c53be7","Type":"ContainerStarted","Data":"cd98bd336c04c843cac0021a3c6ccb9ad0b8cc6832f422533426d8fb10ec55d4"}
Nov 01 06:28:57.281343 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:57.281273    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562" podStartSLOduration=2.281253141 podStartE2EDuration="2.281253141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:57.274899902 +0000 UTC m=+1148.842706733" watchObservedRunningTime="2026-11-01 06:28:57.281253141 +0000 UTC m=+1148.849059972"
Nov 01 06:28:58.171324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:58.171286    4468 generic.go:356] "Generic (PLEG): container finished" podID="b0acf3f5-6233-422f-b186-96d1a3c53be7" containerID="fb4c6872cab1de07b69b72c94c6ddb86d391fb9032441a437673e77cbca5d6a9" exitCode=0
Nov 01 06:28:58.171762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:58.171368    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562" event={"ID":"b0acf3f5-6233-422f-b186-96d1a3c53be7","Type":"ContainerDied","Data":"fb4c6872cab1de07b69b72c94c6ddb86d391fb9032441a437673e77cbca5d6a9"}
Nov 01 06:28:59.440080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.439662    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:28:59.525970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.524942    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2\" (UniqueName: \"kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2\") pod \"b0acf3f5-6233-422f-b186-96d1a3c53be7\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") "
Nov 01 06:28:59.525970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.525025    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca\") pod \"b0acf3f5-6233-422f-b186-96d1a3c53be7\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") "
Nov 01 06:28:59.525970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.525223    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca\") pod \"b0acf3f5-6233-422f-b186-96d1a3c53be7\" (UID: \"b0acf3f5-6233-422f-b186-96d1a3c53be7\") "
Nov 01 06:28:59.526802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.526442    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca" pod "b0acf3f5-6233-422f-b186-96d1a3c53be7" (UID: "b0acf3f5-6233-422f-b186-96d1a3c53be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:59.549953 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.549907    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2" pod "b0acf3f5-6233-422f-b186-96d1a3c53be7" (UID: "b0acf3f5-6233-422f-b186-96d1a3c53be7"). InnerVolumeSpecName "kube-api-access-2lnt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:59.550211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.550176    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca" pod "b0acf3f5-6233-422f-b186-96d1a3c53be7" (UID: "b0acf3f5-6233-422f-b186-96d1a3c53be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:28:59.648501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.629446    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:59.648501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.642615    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lnt2\" (UniqueName: \"kubernetes.io/projected/b0acf3f5-6233-422f-b186-96d1a3c53be7-kube-api-access-2lnt2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:59.648501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:28:59.642644    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0acf3f5-6233-422f-b186-96d1a3c53be7-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:00.273489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:00.273034    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562" event={"ID":"b0acf3f5-6233-422f-b186-96d1a3c53be7","Type":"ContainerDied","Data":"cd98bd336c04c843cac0021a3c6ccb9ad0b8cc6832f422533426d8fb10ec55d4"}
Nov 01 06:29:00.273489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:00.273117    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd98bd336c04c843cac0021a3c6ccb9ad0b8cc6832f422533426d8fb10ec55d4"
Nov 01 06:29:00.273489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:00.273207    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"
Nov 01 06:29:03.011913 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:29:03.011872    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:29:06.505764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.505726    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"]
Nov 01 06:29:06.506310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.506297    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b0acf3f5-6233-422f-b186-96d1a3c53be7" containerName="revoke-keys"
Nov 01 06:29:06.506353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.506311    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0acf3f5-6233-422f-b186-96d1a3c53be7" containerName="revoke-keys"
Nov 01 06:29:06.506414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.506405    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0acf3f5-6233-422f-b186-96d1a3c53be7" containerName="revoke-keys"
Nov 01 06:29:06.509854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.509838    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.562261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.562232    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:29:06.585335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.585300    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvs9v\" (UniqueName: \"kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.585528 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.585359    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.585528 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.585503    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.611758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.611721    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:29:06.611758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.611743    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:29:06.611948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.611815    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:29:06.615678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.615657    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"]
Nov 01 06:29:06.687179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.687150    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.687429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.687412    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvs9v\" (UniqueName: \"kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.687547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.687483    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.687894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.687874    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.690280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.690258    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.743827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.743797    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvs9v\" (UniqueName: \"kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v\") pod \"maas-api-revoke-sub-7e21ccfc-qhlhc\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:06.822535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:06.822508    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:07.248427 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:29:07.248399    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa42faa_aa32_4d8b_a46f_cdd186278093.slice/crio-76d610a2d780859a43f2b80a0e9e2c33c91610ab5d7a5d0c30d5c8512381048e WatchSource:0}: Error finding container 76d610a2d780859a43f2b80a0e9e2c33c91610ab5d7a5d0c30d5c8512381048e: Status 404 returned error can't find the container with id 76d610a2d780859a43f2b80a0e9e2c33c91610ab5d7a5d0c30d5c8512381048e
Nov 01 06:29:07.248561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:07.248460    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"]
Nov 01 06:29:07.326729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:07.326699    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc" event={"ID":"2fa42faa-aa32-4d8b-a46f-cdd186278093","Type":"ContainerStarted","Data":"76d610a2d780859a43f2b80a0e9e2c33c91610ab5d7a5d0c30d5c8512381048e"}
Nov 01 06:29:08.335402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:08.335368    4468 generic.go:356] "Generic (PLEG): container finished" podID="2fa42faa-aa32-4d8b-a46f-cdd186278093" containerID="82740c020a8264987df6d43edd2989833faa37c13ff7b37c2b3d23db7826a123" exitCode=0
Nov 01 06:29:08.335802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:08.335410    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc" event={"ID":"2fa42faa-aa32-4d8b-a46f-cdd186278093","Type":"ContainerDied","Data":"82740c020a8264987df6d43edd2989833faa37c13ff7b37c2b3d23db7826a123"}
Nov 01 06:29:09.493156 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.493133    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:09.625826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.625740    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca\") pod \"2fa42faa-aa32-4d8b-a46f-cdd186278093\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") "
Nov 01 06:29:09.625999 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.625884    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca\") pod \"2fa42faa-aa32-4d8b-a46f-cdd186278093\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") "
Nov 01 06:29:09.625999 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.625915    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v\" (UniqueName: \"kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v\") pod \"2fa42faa-aa32-4d8b-a46f-cdd186278093\" (UID: \"2fa42faa-aa32-4d8b-a46f-cdd186278093\") "
Nov 01 06:29:09.626145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.626122    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca" pod "2fa42faa-aa32-4d8b-a46f-cdd186278093" (UID: "2fa42faa-aa32-4d8b-a46f-cdd186278093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:29:09.626587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.626567    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:09.628267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.628241    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v" pod "2fa42faa-aa32-4d8b-a46f-cdd186278093" (UID: "2fa42faa-aa32-4d8b-a46f-cdd186278093"). InnerVolumeSpecName "kube-api-access-vvs9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:29:09.628524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.628508    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca" pod "2fa42faa-aa32-4d8b-a46f-cdd186278093" (UID: "2fa42faa-aa32-4d8b-a46f-cdd186278093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:09.727561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.727533    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa42faa-aa32-4d8b-a46f-cdd186278093-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:09.727561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:09.727557    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvs9v\" (UniqueName: \"kubernetes.io/projected/2fa42faa-aa32-4d8b-a46f-cdd186278093-kube-api-access-vvs9v\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:10.350628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:10.350595    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"
Nov 01 06:29:10.350790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:10.350601    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc" event={"ID":"2fa42faa-aa32-4d8b-a46f-cdd186278093","Type":"ContainerDied","Data":"76d610a2d780859a43f2b80a0e9e2c33c91610ab5d7a5d0c30d5c8512381048e"}
Nov 01 06:29:10.350790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:10.350725    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d610a2d780859a43f2b80a0e9e2c33c91610ab5d7a5d0c30d5c8512381048e"
Nov 01 06:29:16.047189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.047155    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"]
Nov 01 06:29:16.097689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.097655    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f303bf6a-25cpr"]
Nov 01 06:29:16.590190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.590163    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976fcff3-f87a-40f0-97ce-acbc87b90a3d" path="/var/lib/kubelet/pods/976fcff3-f87a-40f0-97ce-acbc87b90a3d/volumes"
Nov 01 06:29:16.651417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.651386    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"]
Nov 01 06:29:16.652023 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.652008    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2fa42faa-aa32-4d8b-a46f-cdd186278093" containerName="revoke-keys"
Nov 01 06:29:16.652068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.652026    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa42faa-aa32-4d8b-a46f-cdd186278093" containerName="revoke-keys"
Nov 01 06:29:16.652151 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.652142    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa42faa-aa32-4d8b-a46f-cdd186278093" containerName="revoke-keys"
Nov 01 06:29:16.656692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.656676    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.712941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.712915    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"]
Nov 01 06:29:16.757203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.757177    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:29:16.757340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.757223    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:29:16.757340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.757243    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:29:16.757340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.757223    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:29:16.760950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.760919    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5phkb\" (UniqueName: \"kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.761044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.760969    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.761188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.761168    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.862099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.862022    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.862252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.862152    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5phkb\" (UniqueName: \"kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.862252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.862210    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.862891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.862864    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.864537 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.864509    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.918758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.918730    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5phkb\" (UniqueName: \"kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb\") pod \"maas-api-revoke-sub-ec319567-vj22n\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:16.970305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:16.970282    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:17.344546 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:29:17.344517    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89992b30_590b_46f0_9b47_602062b4fc10.slice/crio-4c656cc4be2e39916e7a9f9af0d5694f8bd1f6b31638ab76a2068c8e011d10e1 WatchSource:0}: Error finding container 4c656cc4be2e39916e7a9f9af0d5694f8bd1f6b31638ab76a2068c8e011d10e1: Status 404 returned error can't find the container with id 4c656cc4be2e39916e7a9f9af0d5694f8bd1f6b31638ab76a2068c8e011d10e1
Nov 01 06:29:17.344921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:17.344569    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"]
Nov 01 06:29:17.416640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:17.416599    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n" event={"ID":"89992b30-590b-46f0-9b47-602062b4fc10","Type":"ContainerStarted","Data":"4c656cc4be2e39916e7a9f9af0d5694f8bd1f6b31638ab76a2068c8e011d10e1"}
Nov 01 06:29:18.426445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:18.426304    4468 generic.go:356] "Generic (PLEG): container finished" podID="89992b30-590b-46f0-9b47-602062b4fc10" containerID="b40f72362c23a62c9313eca690a81611fb325e1c0ec185026a8e2b00dc7801a8" exitCode=0
Nov 01 06:29:18.426445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:18.426345    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n" event={"ID":"89992b30-590b-46f0-9b47-602062b4fc10","Type":"ContainerDied","Data":"b40f72362c23a62c9313eca690a81611fb325e1c0ec185026a8e2b00dc7801a8"}
Nov 01 06:29:19.673591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.673560    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:19.802919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.802885    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca\") pod \"89992b30-590b-46f0-9b47-602062b4fc10\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") "
Nov 01 06:29:19.803332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.803308    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca\") pod \"89992b30-590b-46f0-9b47-602062b4fc10\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") "
Nov 01 06:29:19.803581 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.803563    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb\" (UniqueName: \"kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb\") pod \"89992b30-590b-46f0-9b47-602062b4fc10\" (UID: \"89992b30-590b-46f0-9b47-602062b4fc10\") "
Nov 01 06:29:19.805552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.805526    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca" pod "89992b30-590b-46f0-9b47-602062b4fc10" (UID: "89992b30-590b-46f0-9b47-602062b4fc10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:29:19.813386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.813354    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb" pod "89992b30-590b-46f0-9b47-602062b4fc10" (UID: "89992b30-590b-46f0-9b47-602062b4fc10"). InnerVolumeSpecName "kube-api-access-5phkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:29:19.814056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.814024    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca" pod "89992b30-590b-46f0-9b47-602062b4fc10" (UID: "89992b30-590b-46f0-9b47-602062b4fc10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:19.905981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.905947    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5phkb\" (UniqueName: \"kubernetes.io/projected/89992b30-590b-46f0-9b47-602062b4fc10-kube-api-access-5phkb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:19.905981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.905979    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:19.905981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:19.905989    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89992b30-590b-46f0-9b47-602062b4fc10-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:20.459423 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:20.459395    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"
Nov 01 06:29:20.459423 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:20.459418    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n" event={"ID":"89992b30-590b-46f0-9b47-602062b4fc10","Type":"ContainerDied","Data":"4c656cc4be2e39916e7a9f9af0d5694f8bd1f6b31638ab76a2068c8e011d10e1"}
Nov 01 06:29:20.459734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:20.459501    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c656cc4be2e39916e7a9f9af0d5694f8bd1f6b31638ab76a2068c8e011d10e1"
Nov 01 06:29:25.569042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:25.569007    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:29:28.102511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:28.101614    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:29:28.202502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:28.196542    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx" podUID="82915854-11e3-466d-a38e-2f235096f61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:29:28.929493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:28.929409    4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:29:29.110422 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:29.107562    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:29:29.110422 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:29.109500    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:29:29.466653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:29.466596    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:29:29.558498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:29.557961    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 06:29:32.948597 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:29:32.948549    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:29:33.912136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.912099    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"]
Nov 01 06:29:33.912701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.912685    4468 cpu_manager.go:414] "containerMap: removing container" podUID="89992b30-590b-46f0-9b47-602062b4fc10" containerName="revoke-keys"
Nov 01 06:29:33.912764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.912702    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89992b30-590b-46f0-9b47-602062b4fc10" containerName="revoke-keys"
Nov 01 06:29:33.912815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.912806    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="89992b30-590b-46f0-9b47-602062b4fc10" containerName="revoke-keys"
Nov 01 06:29:33.918065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.918046    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:33.972757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.972720    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"]
Nov 01 06:29:33.983243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.983206    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:33.983424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.983365    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:33.983529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:33.983510    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcpq\" (UniqueName: \"kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.016711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.016676    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:29:34.016894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.016786    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:29:34.016894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.016824    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:29:34.019586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.019571    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:29:34.084850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.084814    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.085052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.084884    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.085052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.084966    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcpq\" (UniqueName: \"kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.086092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.086064    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.087392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.087375    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.142507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.142449    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcpq\" (UniqueName: \"kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq\") pod \"maas-api-revoke-sub-dbc6dc2c-sjmvj\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.236272 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.236174    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:34.618846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.618814    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"]
Nov 01 06:29:34.619149 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:29:34.619129    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96280be4_3a6c_4f5c_bdae_87316a0e5e92.slice/crio-a013d51cb1628dae94c9c9174656c44a8e5afef1f7a19839ab1cf46f65421938 WatchSource:0}: Error finding container a013d51cb1628dae94c9c9174656c44a8e5afef1f7a19839ab1cf46f65421938: Status 404 returned error can't find the container with id a013d51cb1628dae94c9c9174656c44a8e5afef1f7a19839ab1cf46f65421938
Nov 01 06:29:34.772829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:34.772794    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj" event={"ID":"96280be4-3a6c-4f5c-bdae-87316a0e5e92","Type":"ContainerStarted","Data":"a013d51cb1628dae94c9c9174656c44a8e5afef1f7a19839ab1cf46f65421938"}
Nov 01 06:29:35.780920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:35.780842    4468 generic.go:356] "Generic (PLEG): container finished" podID="96280be4-3a6c-4f5c-bdae-87316a0e5e92" containerID="d4f43c69c56a3112d46fdff8604931dfc23e53a0b84c3533b9b91eae3335ee51" exitCode=0
Nov 01 06:29:35.780920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:35.780882    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj" event={"ID":"96280be4-3a6c-4f5c-bdae-87316a0e5e92","Type":"ContainerDied","Data":"d4f43c69c56a3112d46fdff8604931dfc23e53a0b84c3533b9b91eae3335ee51"}
Nov 01 06:29:36.789559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.789525    4468 generic.go:356] "Generic (PLEG): container finished" podID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerID="dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2" exitCode=0
Nov 01 06:29:36.789998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.789596    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" event={"ID":"b48a655c-a6fe-4e33-b42f-e79b37b7c142","Type":"ContainerDied","Data":"dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"}
Nov 01 06:29:36.790104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.790074    4468 scope.go:122] "RemoveContainer" containerID="dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"
Nov 01 06:29:36.791571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.791536    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:29:36.791682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.791626    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:29:36.791752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.791682    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:29:36.968915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:36.968886    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:37.020978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.020942    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq\" (UniqueName: \"kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq\") pod \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") "
Nov 01 06:29:37.021176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.020984    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca\") pod \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") "
Nov 01 06:29:37.021248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.021191    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca\") pod \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\" (UID: \"96280be4-3a6c-4f5c-bdae-87316a0e5e92\") "
Nov 01 06:29:37.021549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.021521    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca" pod "96280be4-3a6c-4f5c-bdae-87316a0e5e92" (UID: "96280be4-3a6c-4f5c-bdae-87316a0e5e92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:29:37.021824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.021684    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:37.023311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.023284    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca" pod "96280be4-3a6c-4f5c-bdae-87316a0e5e92" (UID: "96280be4-3a6c-4f5c-bdae-87316a0e5e92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:37.023445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.023429    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq" pod "96280be4-3a6c-4f5c-bdae-87316a0e5e92" (UID: "96280be4-3a6c-4f5c-bdae-87316a0e5e92"). InnerVolumeSpecName "kube-api-access-6tcpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:29:37.122917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.122886    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tcpq\" (UniqueName: \"kubernetes.io/projected/96280be4-3a6c-4f5c-bdae-87316a0e5e92-kube-api-access-6tcpq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:37.122917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.122915    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96280be4-3a6c-4f5c-bdae-87316a0e5e92-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:37.798105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.798066    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"
Nov 01 06:29:37.798105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.798077    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj" event={"ID":"96280be4-3a6c-4f5c-bdae-87316a0e5e92","Type":"ContainerDied","Data":"a013d51cb1628dae94c9c9174656c44a8e5afef1f7a19839ab1cf46f65421938"}
Nov 01 06:29:37.798685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.798132    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a013d51cb1628dae94c9c9174656c44a8e5afef1f7a19839ab1cf46f65421938"
Nov 01 06:29:37.800760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:37.800730    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" event={"ID":"b48a655c-a6fe-4e33-b42f-e79b37b7c142","Type":"ContainerStarted","Data":"fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1"}
Nov 01 06:29:38.121552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:38.109932    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:29:38.121552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:38.110093    4468 patch_prober.go:29] interesting pod/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 06:29:38.121552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:38.110138    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 06:29:39.102835 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:39.102798    4468 patch_prober.go:29] interesting pod/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 06:29:39.103494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:39.103431    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 06:29:40.104963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:40.104923    4468 patch_prober.go:29] interesting pod/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 06:29:40.105635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:40.105593    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 06:29:41.069043 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:41.069001    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"]
Nov 01 06:29:41.103893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:41.103847    4468 patch_prober.go:29] interesting pod/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 06:29:41.104108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:41.103915    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 06:29:41.104386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:41.104362    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d53aedf-2zmtg"]
Nov 01 06:29:42.122497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:42.111872    4468 patch_prober.go:29] interesting pod/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 06:29:42.122497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:42.111952    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 06:29:42.590636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:42.590604    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a916d09-6233-4ebc-aaa4-c159222fea79" path="/var/lib/kubelet/pods/2a916d09-6233-4ebc-aaa4-c159222fea79/volumes"
Nov 01 06:29:42.724552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:42.723582    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:29:43.123493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:43.110893    4468 patch_prober.go:29] interesting pod/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 06:29:43.123493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:43.110951    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 06:29:44.108011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.107980    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:29:44.109299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.109273    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:29:44.670599 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.670558    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"]
Nov 01 06:29:44.671147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.671133    4468 cpu_manager.go:414] "containerMap: removing container" podUID="96280be4-3a6c-4f5c-bdae-87316a0e5e92" containerName="revoke-keys"
Nov 01 06:29:44.671197 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.671148    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96280be4-3a6c-4f5c-bdae-87316a0e5e92" containerName="revoke-keys"
Nov 01 06:29:44.671281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.671273    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="96280be4-3a6c-4f5c-bdae-87316a0e5e92" containerName="revoke-keys"
Nov 01 06:29:44.675997 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.675981    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.717569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.717526    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.717790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.717769    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzf9\" (UniqueName: \"kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.717852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.717840    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.728665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.728635    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:29:44.728665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.728670    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:29:44.728913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.728674    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:29:44.731016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.730991    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"]
Nov 01 06:29:44.776077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.776049    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:29:44.819628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.819600    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzf9\" (UniqueName: \"kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.819811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.819654    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.819811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.819692    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.820349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.820325    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.822242 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.822220    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.876278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.876243    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzf9\" (UniqueName: \"kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9\") pod \"maas-api-revoke-sub-23734bcc-7qmt2\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:44.989956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:44.989884    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:45.047013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.046896    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"]
Nov 01 06:29:45.050836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.050808    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d0b8fa-5gzrv"]
Nov 01 06:29:45.088963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.088920    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"]
Nov 01 06:29:45.096757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.096730    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.148250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.148219    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"]
Nov 01 06:29:45.148445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.148418    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b354e3\""
Nov 01 06:29:45.148445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.148434    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b354e3-dockercfg-wvs8q\""
Nov 01 06:29:45.224339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.224307    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.224533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.224346    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.325367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.325331    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.325569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.325373    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.326014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.325995    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.368007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.367976    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"]
Nov 01 06:29:45.368697 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:29:45.368669    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99756e7_4423_40ec_91d3_66be07388017.slice/crio-35ef251dd879cdf3222507f1bf71eea45023adf92d4e8fee6528a7fd65dbddb2 WatchSource:0}: Error finding container 35ef251dd879cdf3222507f1bf71eea45023adf92d4e8fee6528a7fd65dbddb2: Status 404 returned error can't find the container with id 35ef251dd879cdf3222507f1bf71eea45023adf92d4e8fee6528a7fd65dbddb2
Nov 01 06:29:45.381640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.381615    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.411395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.411370    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.786448 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.786411    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"]
Nov 01 06:29:45.786826 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:29:45.786582    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6df96d_ad9c_4962_8c36_8ed0cdfab191.slice/crio-8c0ad91cf9b2982e6587a7f3e112914c5a1fe78ae5d0465c014bc9e6b6ca0cf4 WatchSource:0}: Error finding container 8c0ad91cf9b2982e6587a7f3e112914c5a1fe78ae5d0465c014bc9e6b6ca0cf4: Status 404 returned error can't find the container with id 8c0ad91cf9b2982e6587a7f3e112914c5a1fe78ae5d0465c014bc9e6b6ca0cf4
Nov 01 06:29:45.898900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.898863    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"]
Nov 01 06:29:45.903907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.903882    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:45.958855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.958824    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"]
Nov 01 06:29:45.986367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.986332    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq" event={"ID":"ba6df96d-ad9c-4962-8c36-8ed0cdfab191","Type":"ContainerStarted","Data":"8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9"}
Nov 01 06:29:45.986586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.986390    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq" event={"ID":"ba6df96d-ad9c-4962-8c36-8ed0cdfab191","Type":"ContainerStarted","Data":"8c0ad91cf9b2982e6587a7f3e112914c5a1fe78ae5d0465c014bc9e6b6ca0cf4"}
Nov 01 06:29:45.986586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.986417    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:45.988223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.988196    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2" event={"ID":"c99756e7-4423-40ec-91d3-66be07388017","Type":"ContainerStarted","Data":"e883e4d41dd24c3192958acf36b05b9a29e4cca960b34ab28eedb95d2ee3ea0a"}
Nov 01 06:29:45.988356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:45.988240    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2" event={"ID":"c99756e7-4423-40ec-91d3-66be07388017","Type":"ContainerStarted","Data":"35ef251dd879cdf3222507f1bf71eea45023adf92d4e8fee6528a7fd65dbddb2"}
Nov 01 06:29:46.033441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.033408    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.033441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.033448    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xzz\" (UniqueName: \"kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz\") pod \"payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.130401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.130301    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2" podStartSLOduration=2.1302868950000002 podStartE2EDuration="2.130286895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:46.127897041 +0000 UTC m=+1197.695703872" watchObservedRunningTime="2026-11-01 06:29:46.130286895 +0000 UTC m=+1197.698093725"
Nov 01 06:29:46.134969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.134938    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.134969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.134977    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xzz\" (UniqueName: \"kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz\") pod \"payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.135844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.135821    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.192252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.192218    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xzz\" (UniqueName: \"kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz\") pod \"payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.219870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.219837    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:46.242348 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.242289    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq" podStartSLOduration=1.242270366 podStartE2EDuration="1.242270366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:46.241043212 +0000 UTC m=+1197.808850043" watchObservedRunningTime="2026-11-01 06:29:46.242270366 +0000 UTC m=+1197.810077199"
Nov 01 06:29:46.591336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.591304    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed882d7f-00d9-42f4-beee-e57bf1f090d9" path="/var/lib/kubelet/pods/ed882d7f-00d9-42f4-beee-e57bf1f090d9/volumes"
Nov 01 06:29:46.605191 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:29:46.605159    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a783741_283e_4888_9ec0_2d5aa66d8607.slice/crio-133433fe57ac9aa518e52d72662b346da8d6abed51e76be287d81b442acbc58a WatchSource:0}: Error finding container 133433fe57ac9aa518e52d72662b346da8d6abed51e76be287d81b442acbc58a: Status 404 returned error can't find the container with id 133433fe57ac9aa518e52d72662b346da8d6abed51e76be287d81b442acbc58a
Nov 01 06:29:46.605948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.605917    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"]
Nov 01 06:29:46.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.998577    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x" event={"ID":"0a783741-283e-4888-9ec0-2d5aa66d8607","Type":"ContainerStarted","Data":"70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa"}
Nov 01 06:29:46.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.998636    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x" event={"ID":"0a783741-283e-4888-9ec0-2d5aa66d8607","Type":"ContainerStarted","Data":"133433fe57ac9aa518e52d72662b346da8d6abed51e76be287d81b442acbc58a"}
Nov 01 06:29:46.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:46.998665    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:47.000444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:47.000415    4468 generic.go:356] "Generic (PLEG): container finished" podID="c99756e7-4423-40ec-91d3-66be07388017" containerID="e883e4d41dd24c3192958acf36b05b9a29e4cca960b34ab28eedb95d2ee3ea0a" exitCode=0
Nov 01 06:29:47.000609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:47.000503    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2" event={"ID":"c99756e7-4423-40ec-91d3-66be07388017","Type":"ContainerDied","Data":"e883e4d41dd24c3192958acf36b05b9a29e4cca960b34ab28eedb95d2ee3ea0a"}
Nov 01 06:29:47.117057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:47.116999    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x" podStartSLOduration=2.116985605 podStartE2EDuration="2.116985605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:47.112640595 +0000 UTC m=+1198.680447429" watchObservedRunningTime="2026-11-01 06:29:47.116985605 +0000 UTC m=+1198.684792467"
Nov 01 06:29:48.156535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.156505    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:48.265958 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.265867    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca\") pod \"c99756e7-4423-40ec-91d3-66be07388017\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") "
Nov 01 06:29:48.266118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.266007    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9\" (UniqueName: \"kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9\") pod \"c99756e7-4423-40ec-91d3-66be07388017\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") "
Nov 01 06:29:48.266118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.266090    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca\") pod \"c99756e7-4423-40ec-91d3-66be07388017\" (UID: \"c99756e7-4423-40ec-91d3-66be07388017\") "
Nov 01 06:29:48.266218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.266197    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca" pod "c99756e7-4423-40ec-91d3-66be07388017" (UID: "c99756e7-4423-40ec-91d3-66be07388017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:29:48.266652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.266631    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99756e7-4423-40ec-91d3-66be07388017-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:48.268517 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.268489    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9" pod "c99756e7-4423-40ec-91d3-66be07388017" (UID: "c99756e7-4423-40ec-91d3-66be07388017"). InnerVolumeSpecName "kube-api-access-cjzf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:29:48.268777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.268750    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca" pod "c99756e7-4423-40ec-91d3-66be07388017" (UID: "c99756e7-4423-40ec-91d3-66be07388017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:48.367704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.367665    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzf9\" (UniqueName: \"kubernetes.io/projected/c99756e7-4423-40ec-91d3-66be07388017-kube-api-access-cjzf9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:48.367704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:48.367702    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99756e7-4423-40ec-91d3-66be07388017-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:49.016563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:49.016457    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2" event={"ID":"c99756e7-4423-40ec-91d3-66be07388017","Type":"ContainerDied","Data":"35ef251dd879cdf3222507f1bf71eea45023adf92d4e8fee6528a7fd65dbddb2"}
Nov 01 06:29:49.016563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:49.016525    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"
Nov 01 06:29:49.016750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:49.016575    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ef251dd879cdf3222507f1bf71eea45023adf92d4e8fee6528a7fd65dbddb2"
Nov 01 06:29:50.038807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:50.038774    4468 scope.go:122] "RemoveContainer" containerID="be4f5e2a3bada792487cda30bd0bb500c802fe2ed1c2a98d1da9e8bc041aee98"
Nov 01 06:29:50.048733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:50.048716    4468 scope.go:122] "RemoveContainer" containerID="f77a99f8c1cc541045041a3ddc594c12468b7426c16503b040090ea1bf2f36a4"
Nov 01 06:29:50.057846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:50.057829    4468 scope.go:122] "RemoveContainer" containerID="ba0a9fe1e67ef234398406389b18b4adf699ce5078581a1dc890d2522929c53b"
Nov 01 06:29:54.975315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:54.975275    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:29:57.005644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:57.005612    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:29:57.059165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:57.059131    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"]
Nov 01 06:29:57.121482 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:57.121423    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-504b94f6-tmm8x"]
Nov 01 06:29:57.363495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:57.331563    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:29:58.031987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:58.031958    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:29:58.591683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:29:58.591641    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b5c367-c1c3-4cc6-8b39-27a8443e98fd" path="/var/lib/kubelet/pods/d9b5c367-c1c3-4cc6-8b39-27a8443e98fd/volumes"
Nov 01 06:30:00.164021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.163936    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"]
Nov 01 06:30:00.164582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.164567    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c99756e7-4423-40ec-91d3-66be07388017" containerName="revoke-keys"
Nov 01 06:30:00.164632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.164582    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99756e7-4423-40ec-91d3-66be07388017" containerName="revoke-keys"
Nov 01 06:30:00.164697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.164687    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99756e7-4423-40ec-91d3-66be07388017" containerName="revoke-keys"
Nov 01 06:30:00.169510 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.169494    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.176342 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.176310    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"]
Nov 01 06:30:00.180826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.180803    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.187665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.187635    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"]
Nov 01 06:30:00.193195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.193169    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.199525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.199496    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"]
Nov 01 06:30:00.204927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.204901    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.215320 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.215293    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"]
Nov 01 06:30:00.219544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.219523    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.222094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.222075    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:30:00.222233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.222173    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:30:00.222233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.222215    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:30:00.271877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.271843    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"]
Nov 01 06:30:00.274377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.274340    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflcm\" (UniqueName: \"kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.274574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.274418    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.274645 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.274619    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.274702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.274680    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9g8\" (UniqueName: \"kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8\") pod \"maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.274764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.274747    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.274869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.274851    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz8h\" (UniqueName: \"kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h\") pod \"maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.276049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.276018    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"]
Nov 01 06:30:00.328627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.328594    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"]
Nov 01 06:30:00.334612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.334580    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"]
Nov 01 06:30:00.370789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.370748    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"]
Nov 01 06:30:00.375853 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.375815    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9g8\" (UniqueName: \"kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8\") pod \"maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.376034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.375909    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9rz\" (UniqueName: \"kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.376034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.375944    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.376034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.375979    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lknk\" (UniqueName: \"kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.376214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376135    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz8h\" (UniqueName: \"kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h\") pod \"maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.376214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376190    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.376316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376225    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.376371 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376314    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflcm\" (UniqueName: \"kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.376429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376369    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.376504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376450    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.376776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.376745    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.377063 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.377044    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.377327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.377304    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.432870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.432803    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9g8\" (UniqueName: \"kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8\") pod \"maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.432870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.432827    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz8h\" (UniqueName: \"kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h\") pod \"maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.433041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.432895    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflcm\" (UniqueName: \"kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.478179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.478146    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.478179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.478184    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.478428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.478330    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9rz\" (UniqueName: \"kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.478428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.478353    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lknk\" (UniqueName: \"kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.478897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.478871    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.479082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.479061    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.483858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.483838    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:00.503549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.503519    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:00.510879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.510853    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:00.535933 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.535881    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lknk\" (UniqueName: \"kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.538409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.538118    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9rz\" (UniqueName: \"kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.821451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.821411    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:00.835131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.835094    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:00.866654 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:00.866617    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3b5262_03f3_434a_813b_3428689cfc98.slice/crio-7aa84c26d6e038a5e351eb67506f168bed58bdc386e399759176c263af6b1fe0 WatchSource:0}: Error finding container 7aa84c26d6e038a5e351eb67506f168bed58bdc386e399759176c263af6b1fe0: Status 404 returned error can't find the container with id 7aa84c26d6e038a5e351eb67506f168bed58bdc386e399759176c263af6b1fe0
Nov 01 06:30:00.867322 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.867291    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"]
Nov 01 06:30:00.920238 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:00.920174    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975b1319_328e_492a_8303_087d150d5d7d.slice/crio-c4b150e09f1c577da3db85d1a67ee630e885ca6b4e055ea0b1740e0fbc4e4aed WatchSource:0}: Error finding container c4b150e09f1c577da3db85d1a67ee630e885ca6b4e055ea0b1740e0fbc4e4aed: Status 404 returned error can't find the container with id c4b150e09f1c577da3db85d1a67ee630e885ca6b4e055ea0b1740e0fbc4e4aed
Nov 01 06:30:00.923243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.921291    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"]
Nov 01 06:30:00.928134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:00.927775    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"]
Nov 01 06:30:01.045176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.045141    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"]
Nov 01 06:30:01.053431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.053401    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aed737-ffflq"]
Nov 01 06:30:01.155207 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.155171    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerStarted","Data":"b27fc10d856ec0ef1b3c0955620d475da28d87b53430eb123ea321611b5e2809"}
Nov 01 06:30:01.156450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.156424    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" event={"ID":"ee3b5262-03f3-434a-813b-3428689cfc98","Type":"ContainerStarted","Data":"7aa84c26d6e038a5e351eb67506f168bed58bdc386e399759176c263af6b1fe0"}
Nov 01 06:30:01.157601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.157582    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerStarted","Data":"c4b150e09f1c577da3db85d1a67ee630e885ca6b4e055ea0b1740e0fbc4e4aed"}
Nov 01 06:30:01.205902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.205873    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"]
Nov 01 06:30:01.206773 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:01.206747    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1c374a_5646_47fd_bcfb_a92a71fda553.slice/crio-8b737f3c071e841efb2c454382fab9eb6d7b5ef24b90214b5beb111d13eea98d WatchSource:0}: Error finding container 8b737f3c071e841efb2c454382fab9eb6d7b5ef24b90214b5beb111d13eea98d: Status 404 returned error can't find the container with id 8b737f3c071e841efb2c454382fab9eb6d7b5ef24b90214b5beb111d13eea98d
Nov 01 06:30:01.224215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:01.224189    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"]
Nov 01 06:30:01.224453 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:01.224429    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807be97f_c7b4_4e67_84a9_c61d4b291410.slice/crio-60c8135e6c7aa4c1ba9183c161d83cd0c11489d38c092973f8f79fbfd7947514 WatchSource:0}: Error finding container 60c8135e6c7aa4c1ba9183c161d83cd0c11489d38c092973f8f79fbfd7947514: Status 404 returned error can't find the container with id 60c8135e6c7aa4c1ba9183c161d83cd0c11489d38c092973f8f79fbfd7947514
Nov 01 06:30:02.168340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.168304    4468 generic.go:356] "Generic (PLEG): container finished" podID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerID="862af9fa3ff029f2ed1c4d6a7644eebdfa2bd276b09ab94605f797db137ffa69" exitCode=60
Nov 01 06:30:02.168573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.168389    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerDied","Data":"862af9fa3ff029f2ed1c4d6a7644eebdfa2bd276b09ab94605f797db137ffa69"}
Nov 01 06:30:02.168775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.168759    4468 scope.go:122] "RemoveContainer" containerID="862af9fa3ff029f2ed1c4d6a7644eebdfa2bd276b09ab94605f797db137ffa69"
Nov 01 06:30:02.170220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.170195    4468 generic.go:356] "Generic (PLEG): container finished" podID="ee3b5262-03f3-434a-813b-3428689cfc98" containerID="8f54e8bf59aac44549233f414660b632e9a8461ec3c3a743213e55193ffbf2de" exitCode=60
Nov 01 06:30:02.170345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.170280    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" event={"ID":"ee3b5262-03f3-434a-813b-3428689cfc98","Type":"ContainerDied","Data":"8f54e8bf59aac44549233f414660b632e9a8461ec3c3a743213e55193ffbf2de"}
Nov 01 06:30:02.170631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.170549    4468 scope.go:122] "RemoveContainer" containerID="8f54e8bf59aac44549233f414660b632e9a8461ec3c3a743213e55193ffbf2de"
Nov 01 06:30:02.172199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.172176    4468 generic.go:356] "Generic (PLEG): container finished" podID="975b1319-328e-492a-8303-087d150d5d7d" containerID="3f5f43c7ec4b626f37499c858733af65ce69766b5eacd9f60261a89002cc71d3" exitCode=60
Nov 01 06:30:02.172364 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.172277    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerDied","Data":"3f5f43c7ec4b626f37499c858733af65ce69766b5eacd9f60261a89002cc71d3"}
Nov 01 06:30:02.172639 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.172623    4468 scope.go:122] "RemoveContainer" containerID="3f5f43c7ec4b626f37499c858733af65ce69766b5eacd9f60261a89002cc71d3"
Nov 01 06:30:02.174135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.174108    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerStarted","Data":"077c8abea3f234b00ad8ff3ecdd53457e308e905a6f8b2cdaf38bd41468dfcff"}
Nov 01 06:30:02.174214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.174158    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerStarted","Data":"8b737f3c071e841efb2c454382fab9eb6d7b5ef24b90214b5beb111d13eea98d"}
Nov 01 06:30:02.175920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.175898    4468 generic.go:356] "Generic (PLEG): container finished" podID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerID="165fad2eb65df4cf696ed11b9f68959dc5c95b51c679876ae3eea4cfd2312e06" exitCode=60
Nov 01 06:30:02.176024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.175931    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerDied","Data":"165fad2eb65df4cf696ed11b9f68959dc5c95b51c679876ae3eea4cfd2312e06"}
Nov 01 06:30:02.176024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.175959    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerStarted","Data":"60c8135e6c7aa4c1ba9183c161d83cd0c11489d38c092973f8f79fbfd7947514"}
Nov 01 06:30:02.176192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.176175    4468 scope.go:122] "RemoveContainer" containerID="165fad2eb65df4cf696ed11b9f68959dc5c95b51c679876ae3eea4cfd2312e06"
Nov 01 06:30:02.596217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.596184    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279f8367-0f05-472c-a761-1ba5f3bcdf29" path="/var/lib/kubelet/pods/279f8367-0f05-472c-a761-1ba5f3bcdf29/volumes"
Nov 01 06:30:02.734804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:02.734681    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" podStartSLOduration=2.734666183 podStartE2EDuration="2.734666183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:02.731955397 +0000 UTC m=+1214.299762225" watchObservedRunningTime="2026-11-01 06:30:02.734666183 +0000 UTC m=+1214.302473012"
Nov 01 06:30:02.949710 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:02.949670    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:30:03.187283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.187244    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerStarted","Data":"b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71"}
Nov 01 06:30:03.189583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.189551    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerStarted","Data":"985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0"}
Nov 01 06:30:03.191513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.191483    4468 generic.go:356] "Generic (PLEG): container finished" podID="ee3b5262-03f3-434a-813b-3428689cfc98" containerID="f33d1c29164187d93829e7f66adea088fed4ab20877951e7dfbf3854b169fa9c" exitCode=60
Nov 01 06:30:03.191672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.191560    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" event={"ID":"ee3b5262-03f3-434a-813b-3428689cfc98","Type":"ContainerDied","Data":"f33d1c29164187d93829e7f66adea088fed4ab20877951e7dfbf3854b169fa9c"}
Nov 01 06:30:03.191672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.191617    4468 scope.go:122] "RemoveContainer" containerID="8f54e8bf59aac44549233f414660b632e9a8461ec3c3a743213e55193ffbf2de"
Nov 01 06:30:03.191954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.191934    4468 scope.go:122] "RemoveContainer" containerID="f33d1c29164187d93829e7f66adea088fed4ab20877951e7dfbf3854b169fa9c"
Nov 01 06:30:03.192222 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:03.192200    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl_odh-ai-gateway-infra(ee3b5262-03f3-434a-813b-3428689cfc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" podUID="ee3b5262-03f3-434a-813b-3428689cfc98"
Nov 01 06:30:03.193927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.193903    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerStarted","Data":"4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4"}
Nov 01 06:30:03.195721 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.195695    4468 generic.go:356] "Generic (PLEG): container finished" podID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerID="077c8abea3f234b00ad8ff3ecdd53457e308e905a6f8b2cdaf38bd41468dfcff" exitCode=60
Nov 01 06:30:03.195826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.195775    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerDied","Data":"077c8abea3f234b00ad8ff3ecdd53457e308e905a6f8b2cdaf38bd41468dfcff"}
Nov 01 06:30:03.195978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.195965    4468 scope.go:122] "RemoveContainer" containerID="077c8abea3f234b00ad8ff3ecdd53457e308e905a6f8b2cdaf38bd41468dfcff"
Nov 01 06:30:03.372092 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:03.372008    4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a35cd3f_145b_402a_9ee8_7aee929a0db4.slice/crio-985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 06:30:03.434346 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.434285    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" podStartSLOduration=3.434266785 podStartE2EDuration="3.434266785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:03.429166002 +0000 UTC m=+1214.996972847" watchObservedRunningTime="2026-11-01 06:30:03.434266785 +0000 UTC m=+1215.002073616"
Nov 01 06:30:03.435122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.435082    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" podStartSLOduration=3.435071752 podStartE2EDuration="3.435071752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:03.317722985 +0000 UTC m=+1214.885529815" watchObservedRunningTime="2026-11-01 06:30:03.435071752 +0000 UTC m=+1215.002878582"
Nov 01 06:30:03.661104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:03.660999    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" podStartSLOduration=3.660984336 podStartE2EDuration="3.660984336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:03.657236257 +0000 UTC m=+1215.225043088" watchObservedRunningTime="2026-11-01 06:30:03.660984336 +0000 UTC m=+1215.228791168"
Nov 01 06:30:04.206677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.206631    4468 generic.go:356] "Generic (PLEG): container finished" podID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerID="b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71" exitCode=60
Nov 01 06:30:04.206817 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.206692    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerDied","Data":"b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71"}
Nov 01 06:30:04.206891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.206815    4468 scope.go:122] "RemoveContainer" containerID="165fad2eb65df4cf696ed11b9f68959dc5c95b51c679876ae3eea4cfd2312e06"
Nov 01 06:30:04.207122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.207089    4468 scope.go:122] "RemoveContainer" containerID="b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71"
Nov 01 06:30:04.207578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:04.207454    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm_odh-ai-gateway-infra(807be97f-c7b4-4e67-84a9-c61d4b291410)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410"
Nov 01 06:30:04.209100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.209074    4468 generic.go:356] "Generic (PLEG): container finished" podID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerID="985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0" exitCode=60
Nov 01 06:30:04.209197 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.209148    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerDied","Data":"985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0"}
Nov 01 06:30:04.209444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.209423    4468 scope.go:122] "RemoveContainer" containerID="985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0"
Nov 01 06:30:04.209775 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:04.209752    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z_odh-ai-gateway-infra(5a35cd3f-145b-402a-9ee8-7aee929a0db4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4"
Nov 01 06:30:04.211190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.211172    4468 scope.go:122] "RemoveContainer" containerID="f33d1c29164187d93829e7f66adea088fed4ab20877951e7dfbf3854b169fa9c"
Nov 01 06:30:04.211452 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:04.211417    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl_odh-ai-gateway-infra(ee3b5262-03f3-434a-813b-3428689cfc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" podUID="ee3b5262-03f3-434a-813b-3428689cfc98"
Nov 01 06:30:04.213192 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.213168    4468 generic.go:356] "Generic (PLEG): container finished" podID="975b1319-328e-492a-8303-087d150d5d7d" containerID="4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4" exitCode=60
Nov 01 06:30:04.213279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.213255    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerDied","Data":"4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4"}
Nov 01 06:30:04.213479 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.213448    4468 scope.go:122] "RemoveContainer" containerID="4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4"
Nov 01 06:30:04.213734 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:04.213707    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh_odh-ai-gateway-infra(975b1319-328e-492a-8303-087d150d5d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" podUID="975b1319-328e-492a-8303-087d150d5d7d"
Nov 01 06:30:04.215293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.215273    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerStarted","Data":"0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"}
Nov 01 06:30:04.215442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.215420    4468 scope.go:122] "RemoveContainer" containerID="0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"
Nov 01 06:30:04.215608 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:04.215592    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j_odh-ai-gateway-infra(0e1c374a-5646-47fd-bcfb-a92a71fda553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553"
Nov 01 06:30:04.277992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.277965    4468 scope.go:122] "RemoveContainer" containerID="862af9fa3ff029f2ed1c4d6a7644eebdfa2bd276b09ab94605f797db137ffa69"
Nov 01 06:30:04.291397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:04.291375    4468 scope.go:122] "RemoveContainer" containerID="3f5f43c7ec4b626f37499c858733af65ce69766b5eacd9f60261a89002cc71d3"
Nov 01 06:30:05.224825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.224795    4468 scope.go:122] "RemoveContainer" containerID="985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0"
Nov 01 06:30:05.225264 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:05.225164    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z_odh-ai-gateway-infra(5a35cd3f-145b-402a-9ee8-7aee929a0db4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4"
Nov 01 06:30:05.226546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.226522    4468 scope.go:122] "RemoveContainer" containerID="4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4"
Nov 01 06:30:05.226858 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:05.226824    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh_odh-ai-gateway-infra(975b1319-328e-492a-8303-087d150d5d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" podUID="975b1319-328e-492a-8303-087d150d5d7d"
Nov 01 06:30:05.228216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.228194    4468 generic.go:356] "Generic (PLEG): container finished" podID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerID="0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f" exitCode=60
Nov 01 06:30:05.228307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.228263    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerDied","Data":"0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"}
Nov 01 06:30:05.228345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.228322    4468 scope.go:122] "RemoveContainer" containerID="077c8abea3f234b00ad8ff3ecdd53457e308e905a6f8b2cdaf38bd41468dfcff"
Nov 01 06:30:05.228646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.228622    4468 scope.go:122] "RemoveContainer" containerID="0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"
Nov 01 06:30:05.228937 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:05.228910    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j_odh-ai-gateway-infra(0e1c374a-5646-47fd-bcfb-a92a71fda553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553"
Nov 01 06:30:05.230385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.230368    4468 scope.go:122] "RemoveContainer" containerID="b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71"
Nov 01 06:30:05.230641 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:05.230621    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm_odh-ai-gateway-infra(807be97f-c7b4-4e67-84a9-c61d4b291410)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410"
Nov 01 06:30:05.681770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:05.681740    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x_0a783741-283e-4888-9ec0-2d5aa66d8607/payload-processing/0.log"
Nov 01 06:30:06.239942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:06.239911    4468 scope.go:122] "RemoveContainer" containerID="0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"
Nov 01 06:30:06.240453 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:06.240207    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j_odh-ai-gateway-infra(0e1c374a-5646-47fd-bcfb-a92a71fda553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553"
Nov 01 06:30:13.584087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:13.584049    4468 scope.go:122] "RemoveContainer" containerID="f33d1c29164187d93829e7f66adea088fed4ab20877951e7dfbf3854b169fa9c"
Nov 01 06:30:14.356162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:14.356132    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" event={"ID":"ee3b5262-03f3-434a-813b-3428689cfc98","Type":"ContainerStarted","Data":"fb9e8624ad46ed4a5f3b25a70335ddfb2e30428cc839c0fe7c1f7da55d163d84"}
Nov 01 06:30:14.471417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:14.471361    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" podStartSLOduration=14.47134626 podStartE2EDuration="14.47134626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:14.467090967 +0000 UTC m=+1226.034897797" watchObservedRunningTime="2026-11-01 06:30:14.47134626 +0000 UTC m=+1226.039153117"
Nov 01 06:30:15.365998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.365965    4468 generic.go:356] "Generic (PLEG): container finished" podID="ee3b5262-03f3-434a-813b-3428689cfc98" containerID="fb9e8624ad46ed4a5f3b25a70335ddfb2e30428cc839c0fe7c1f7da55d163d84" exitCode=60
Nov 01 06:30:15.366392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.366034    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" event={"ID":"ee3b5262-03f3-434a-813b-3428689cfc98","Type":"ContainerDied","Data":"fb9e8624ad46ed4a5f3b25a70335ddfb2e30428cc839c0fe7c1f7da55d163d84"}
Nov 01 06:30:15.366392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.366094    4468 scope.go:122] "RemoveContainer" containerID="f33d1c29164187d93829e7f66adea088fed4ab20877951e7dfbf3854b169fa9c"
Nov 01 06:30:15.366610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.366595    4468 scope.go:122] "RemoveContainer" containerID="fb9e8624ad46ed4a5f3b25a70335ddfb2e30428cc839c0fe7c1f7da55d163d84"
Nov 01 06:30:15.366975 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:15.366954    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl_odh-ai-gateway-infra(ee3b5262-03f3-434a-813b-3428689cfc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" podUID="ee3b5262-03f3-434a-813b-3428689cfc98"
Nov 01 06:30:15.478633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.478599    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"]
Nov 01 06:30:15.583300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.583267    4468 scope.go:122] "RemoveContainer" containerID="0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"
Nov 01 06:30:15.583448 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:15.583348    4468 scope.go:122] "RemoveContainer" containerID="985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0"
Nov 01 06:30:16.045854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.045817    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"]
Nov 01 06:30:16.049756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.049725    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a32142-lh27h"]
Nov 01 06:30:16.376117 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.376077    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerStarted","Data":"9e9655f369904b3b910f38f66756976b3fb6bb71d8fb2133b5b98aae6be66395"}
Nov 01 06:30:16.380132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.380106    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerStarted","Data":"ff3e89c764c2f975bb0d1fa5ba7fcb066a727e4d9fbce20361208e787ce2dd69"}
Nov 01 06:30:16.584028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.583608    4468 scope.go:122] "RemoveContainer" containerID="b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71"
Nov 01 06:30:16.592867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.592840    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53be671d-c05a-46ca-a810-6e08191611b8" path="/var/lib/kubelet/pods/53be671d-c05a-46ca-a810-6e08191611b8/volumes"
Nov 01 06:30:16.654764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.654737    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:16.763574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.763530    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca\") pod \"ee3b5262-03f3-434a-813b-3428689cfc98\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") "
Nov 01 06:30:16.763793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.763779    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8\" (UniqueName: \"kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8\") pod \"ee3b5262-03f3-434a-813b-3428689cfc98\" (UID: \"ee3b5262-03f3-434a-813b-3428689cfc98\") "
Nov 01 06:30:16.763939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.763914    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca" pod "ee3b5262-03f3-434a-813b-3428689cfc98" (UID: "ee3b5262-03f3-434a-813b-3428689cfc98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:16.764246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.764229    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b5262-03f3-434a-813b-3428689cfc98-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:16.766136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.766109    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8" pod "ee3b5262-03f3-434a-813b-3428689cfc98" (UID: "ee3b5262-03f3-434a-813b-3428689cfc98"). InnerVolumeSpecName "kube-api-access-8w9g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:16.865498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:16.865412    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9g8\" (UniqueName: \"kubernetes.io/projected/ee3b5262-03f3-434a-813b-3428689cfc98-kube-api-access-8w9g8\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:17.389842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.389809    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerStarted","Data":"fdc908022d5e6ad4bbd411a04e2fea784d21068a1c9081ede248a0c697bd6389"}
Nov 01 06:30:17.391872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.391845    4468 generic.go:356] "Generic (PLEG): container finished" podID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerID="9e9655f369904b3b910f38f66756976b3fb6bb71d8fb2133b5b98aae6be66395" exitCode=60
Nov 01 06:30:17.392020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.391908    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerDied","Data":"9e9655f369904b3b910f38f66756976b3fb6bb71d8fb2133b5b98aae6be66395"}
Nov 01 06:30:17.392020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.391965    4468 scope.go:122] "RemoveContainer" containerID="985785434d9f48d3d0c7874102b673b9cdaa6876136da9130788143f8f637bc0"
Nov 01 06:30:17.392306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.392286    4468 scope.go:122] "RemoveContainer" containerID="9e9655f369904b3b910f38f66756976b3fb6bb71d8fb2133b5b98aae6be66395"
Nov 01 06:30:17.392622 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:17.392594    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z_odh-ai-gateway-infra(5a35cd3f-145b-402a-9ee8-7aee929a0db4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4"
Nov 01 06:30:17.393651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.393610    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"
Nov 01 06:30:17.393716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.393644    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl" event={"ID":"ee3b5262-03f3-434a-813b-3428689cfc98","Type":"ContainerDied","Data":"7aa84c26d6e038a5e351eb67506f168bed58bdc386e399759176c263af6b1fe0"}
Nov 01 06:30:17.395747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.395723    4468 generic.go:356] "Generic (PLEG): container finished" podID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerID="ff3e89c764c2f975bb0d1fa5ba7fcb066a727e4d9fbce20361208e787ce2dd69" exitCode=60
Nov 01 06:30:17.395864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.395795    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerDied","Data":"ff3e89c764c2f975bb0d1fa5ba7fcb066a727e4d9fbce20361208e787ce2dd69"}
Nov 01 06:30:17.396081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.396066    4468 scope.go:122] "RemoveContainer" containerID="ff3e89c764c2f975bb0d1fa5ba7fcb066a727e4d9fbce20361208e787ce2dd69"
Nov 01 06:30:17.396291 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:17.396273    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j_odh-ai-gateway-infra(0e1c374a-5646-47fd-bcfb-a92a71fda553)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553"
Nov 01 06:30:17.406341 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.406323    4468 scope.go:122] "RemoveContainer" containerID="fb9e8624ad46ed4a5f3b25a70335ddfb2e30428cc839c0fe7c1f7da55d163d84"
Nov 01 06:30:17.416545 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.416519    4468 scope.go:122] "RemoveContainer" containerID="0fb0253693362b780274fb20895f31b7c98dad7d530ba8117f45a98c0ae3839f"
Nov 01 06:30:17.501578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.501525    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"]
Nov 01 06:30:17.584158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.584116    4468 scope.go:122] "RemoveContainer" containerID="4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4"
Nov 01 06:30:17.726291 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.726208    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"]
Nov 01 06:30:17.738347 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.738318    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"]
Nov 01 06:30:17.742037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:17.742013    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-e7b2a3-29891910-b5jjl"]
Nov 01 06:30:18.408054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.408018    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerStarted","Data":"a83217194d65ebe00ffb521188704ed446d41c6c63c133f2ca6bfc5015e92258"}
Nov 01 06:30:18.411804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.411768    4468 generic.go:356] "Generic (PLEG): container finished" podID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerID="fdc908022d5e6ad4bbd411a04e2fea784d21068a1c9081ede248a0c697bd6389" exitCode=60
Nov 01 06:30:18.411804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.411798    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerDied","Data":"fdc908022d5e6ad4bbd411a04e2fea784d21068a1c9081ede248a0c697bd6389"}
Nov 01 06:30:18.411987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.411837    4468 scope.go:122] "RemoveContainer" containerID="b600d0851d08c41c490b857c2e38a736d100b871e361a0812a290055339eab71"
Nov 01 06:30:18.412327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.412304    4468 scope.go:122] "RemoveContainer" containerID="fdc908022d5e6ad4bbd411a04e2fea784d21068a1c9081ede248a0c697bd6389"
Nov 01 06:30:18.412696 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:18.412668    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm_odh-ai-gateway-infra(807be97f-c7b4-4e67-84a9-c61d4b291410)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410"
Nov 01 06:30:18.530522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.529541    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"]
Nov 01 06:30:18.591196 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.591165    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:18.591853 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.591825    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" path="/var/lib/kubelet/pods/ee3b5262-03f3-434a-813b-3428689cfc98/volumes"
Nov 01 06:30:18.689512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.689415    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:18.691861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.691835    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca\") pod \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") "
Nov 01 06:30:18.691999 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.691946    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm\" (UniqueName: \"kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm\") pod \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\" (UID: \"5a35cd3f-145b-402a-9ee8-7aee929a0db4\") "
Nov 01 06:30:18.692224 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.692200    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca" pod "5a35cd3f-145b-402a-9ee8-7aee929a0db4" (UID: "5a35cd3f-145b-402a-9ee8-7aee929a0db4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:18.692433 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.692416    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a35cd3f-145b-402a-9ee8-7aee929a0db4-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:18.694172 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.694154    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm" pod "5a35cd3f-145b-402a-9ee8-7aee929a0db4" (UID: "5a35cd3f-145b-402a-9ee8-7aee929a0db4"). InnerVolumeSpecName "kube-api-access-kflcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:18.793613 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.793573    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca\") pod \"0e1c374a-5646-47fd-bcfb-a92a71fda553\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") "
Nov 01 06:30:18.793812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.793724    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk\" (UniqueName: \"kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk\") pod \"0e1c374a-5646-47fd-bcfb-a92a71fda553\" (UID: \"0e1c374a-5646-47fd-bcfb-a92a71fda553\") "
Nov 01 06:30:18.794000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.793972    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca" pod "0e1c374a-5646-47fd-bcfb-a92a71fda553" (UID: "0e1c374a-5646-47fd-bcfb-a92a71fda553"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:18.794183 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.794166    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflcm\" (UniqueName: \"kubernetes.io/projected/5a35cd3f-145b-402a-9ee8-7aee929a0db4-kube-api-access-kflcm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:18.794233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.794189    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c374a-5646-47fd-bcfb-a92a71fda553-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:18.795909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.795889    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk" pod "0e1c374a-5646-47fd-bcfb-a92a71fda553" (UID: "0e1c374a-5646-47fd-bcfb-a92a71fda553"). InnerVolumeSpecName "kube-api-access-2lknk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:18.895652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:18.895610    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lknk\" (UniqueName: \"kubernetes.io/projected/0e1c374a-5646-47fd-bcfb-a92a71fda553-kube-api-access-2lknk\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:19.420992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.420963    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"
Nov 01 06:30:19.421459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.420985    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j" event={"ID":"0e1c374a-5646-47fd-bcfb-a92a71fda553","Type":"ContainerDied","Data":"8b737f3c071e841efb2c454382fab9eb6d7b5ef24b90214b5beb111d13eea98d"}
Nov 01 06:30:19.421459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.421086    4468 scope.go:122] "RemoveContainer" containerID="ff3e89c764c2f975bb0d1fa5ba7fcb066a727e4d9fbce20361208e787ce2dd69"
Nov 01 06:30:19.424914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.424882    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z" event={"ID":"5a35cd3f-145b-402a-9ee8-7aee929a0db4","Type":"ContainerDied","Data":"b27fc10d856ec0ef1b3c0955620d475da28d87b53430eb123ea321611b5e2809"}
Nov 01 06:30:19.425046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.424926    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"
Nov 01 06:30:19.427806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.427776    4468 generic.go:356] "Generic (PLEG): container finished" podID="975b1319-328e-492a-8303-087d150d5d7d" containerID="a83217194d65ebe00ffb521188704ed446d41c6c63c133f2ca6bfc5015e92258" exitCode=60
Nov 01 06:30:19.427910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.427828    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerDied","Data":"a83217194d65ebe00ffb521188704ed446d41c6c63c133f2ca6bfc5015e92258"}
Nov 01 06:30:19.429166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.428532    4468 scope.go:122] "RemoveContainer" containerID="a83217194d65ebe00ffb521188704ed446d41c6c63c133f2ca6bfc5015e92258"
Nov 01 06:30:19.429853 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:19.429821    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh_odh-ai-gateway-infra(975b1319-328e-492a-8303-087d150d5d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" podUID="975b1319-328e-492a-8303-087d150d5d7d"
Nov 01 06:30:19.434537 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.434517    4468 scope.go:122] "RemoveContainer" containerID="9e9655f369904b3b910f38f66756976b3fb6bb71d8fb2133b5b98aae6be66395"
Nov 01 06:30:19.445518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.445452    4468 scope.go:122] "RemoveContainer" containerID="4d142e63c87c1e59d862eec402017e8f95f0ca2e5407975da5da9ee22a43d9d4"
Nov 01 06:30:19.588295 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.588264    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"]
Nov 01 06:30:19.592311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.592277    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891910-d2t7j"]
Nov 01 06:30:19.593160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.593144    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:19.650050 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.650014    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"]
Nov 01 06:30:19.698899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.698802    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"]
Nov 01 06:30:19.704575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.704546    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca\") pod \"807be97f-c7b4-4e67-84a9-c61d4b291410\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") "
Nov 01 06:30:19.704746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.704547    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd9d24-29891910-2cq9z"]
Nov 01 06:30:19.704746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.704727    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz\" (UniqueName: \"kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz\") pod \"807be97f-c7b4-4e67-84a9-c61d4b291410\" (UID: \"807be97f-c7b4-4e67-84a9-c61d4b291410\") "
Nov 01 06:30:19.704848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.704826    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca" pod "807be97f-c7b4-4e67-84a9-c61d4b291410" (UID: "807be97f-c7b4-4e67-84a9-c61d4b291410"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:19.705076 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.705059    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/807be97f-c7b4-4e67-84a9-c61d4b291410-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:19.707069 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.707043    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz" pod "807be97f-c7b4-4e67-84a9-c61d4b291410" (UID: "807be97f-c7b4-4e67-84a9-c61d4b291410"). InnerVolumeSpecName "kube-api-access-9q9rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:19.806354 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:19.806315    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9rz\" (UniqueName: \"kubernetes.io/projected/807be97f-c7b4-4e67-84a9-c61d4b291410-kube-api-access-9q9rz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.436160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.436130    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"
Nov 01 06:30:20.436700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.436120    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm" event={"ID":"807be97f-c7b4-4e67-84a9-c61d4b291410","Type":"ContainerDied","Data":"60c8135e6c7aa4c1ba9183c161d83cd0c11489d38c092973f8f79fbfd7947514"}
Nov 01 06:30:20.436700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.436276    4468 scope.go:122] "RemoveContainer" containerID="fdc908022d5e6ad4bbd411a04e2fea784d21068a1c9081ede248a0c697bd6389"
Nov 01 06:30:20.599050 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.599021    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" path="/var/lib/kubelet/pods/0e1c374a-5646-47fd-bcfb-a92a71fda553/volumes"
Nov 01 06:30:20.600033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.599664    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" path="/var/lib/kubelet/pods/5a35cd3f-145b-402a-9ee8-7aee929a0db4/volumes"
Nov 01 06:30:20.625482 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.625433    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:20.717330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.717226    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h\" (UniqueName: \"kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h\") pod \"975b1319-328e-492a-8303-087d150d5d7d\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") "
Nov 01 06:30:20.717534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.717376    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca\") pod \"975b1319-328e-492a-8303-087d150d5d7d\" (UID: \"975b1319-328e-492a-8303-087d150d5d7d\") "
Nov 01 06:30:20.717852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.717816    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca" pod "975b1319-328e-492a-8303-087d150d5d7d" (UID: "975b1319-328e-492a-8303-087d150d5d7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:20.718027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.718011    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975b1319-328e-492a-8303-087d150d5d7d-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:20.719290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.719272    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"]
Nov 01 06:30:20.719734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.719712    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h" pod "975b1319-328e-492a-8303-087d150d5d7d" (UID: "975b1319-328e-492a-8303-087d150d5d7d"). InnerVolumeSpecName "kube-api-access-5zz8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:20.724366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.724338    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891910-zhfhm"]
Nov 01 06:30:20.819387 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:20.819353    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz8h\" (UniqueName: \"kubernetes.io/projected/975b1319-328e-492a-8303-087d150d5d7d-kube-api-access-5zz8h\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:21.451259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:21.451227    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"
Nov 01 06:30:21.451259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:21.451250    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh" event={"ID":"975b1319-328e-492a-8303-087d150d5d7d","Type":"ContainerDied","Data":"c4b150e09f1c577da3db85d1a67ee630e885ca6b4e055ea0b1740e0fbc4e4aed"}
Nov 01 06:30:21.451847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:21.451322    4468 scope.go:122] "RemoveContainer" containerID="a83217194d65ebe00ffb521188704ed446d41c6c63c133f2ca6bfc5015e92258"
Nov 01 06:30:21.615579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:21.615543    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"]
Nov 01 06:30:21.664667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:21.664627    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b354e3-29891910-kgcvh"]
Nov 01 06:30:22.591025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:22.590988    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" path="/var/lib/kubelet/pods/807be97f-c7b4-4e67-84a9-c61d4b291410/volumes"
Nov 01 06:30:22.591511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:22.591492    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975b1319-328e-492a-8303-087d150d5d7d" path="/var/lib/kubelet/pods/975b1319-328e-492a-8303-087d150d5d7d/volumes"
Nov 01 06:30:23.141216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:23.141180    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:30:25.082734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:25.082686    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"]
Nov 01 06:30:25.123904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:25.123866    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e305fb93-lnjgp"]
Nov 01 06:30:26.193020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:26.192947    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx" podUID="82915854-11e3-466d-a38e-2f235096f61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:30:26.590871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:26.590840    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa07492e-b39d-49b3-bc47-ff259a427355" path="/var/lib/kubelet/pods/fa07492e-b39d-49b3-bc47-ff259a427355/volumes"
Nov 01 06:30:27.567938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:27.567906    4468 generic.go:356] "Generic (PLEG): container finished" podID="388bf2ac-33b8-40c2-a4c1-085dc435a3fa" containerID="16eee90fe72de21487b0acf66e4abf8f36035ffc1ed64b0e479fc13644e3b320" exitCode=0
Nov 01 06:30:27.568337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:27.567979    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw" event={"ID":"388bf2ac-33b8-40c2-a4c1-085dc435a3fa","Type":"ContainerDied","Data":"16eee90fe72de21487b0acf66e4abf8f36035ffc1ed64b0e479fc13644e3b320"}
Nov 01 06:30:28.736390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.736353    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:30:28.828529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.828436    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle\") pod \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") "
Nov 01 06:30:28.828529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.828517    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j\" (UniqueName: \"kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j\") pod \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") "
Nov 01 06:30:28.828760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.828743    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path\") pod \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") "
Nov 01 06:30:28.828844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.828814    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle\") pod \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\" (UID: \"388bf2ac-33b8-40c2-a4c1-085dc435a3fa\") "
Nov 01 06:30:28.828901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.828814    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle" pod "388bf2ac-33b8-40c2-a4c1-085dc435a3fa" (UID: "388bf2ac-33b8-40c2-a4c1-085dc435a3fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:28.829106 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.829080    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path" pod "388bf2ac-33b8-40c2-a4c1-085dc435a3fa" (UID: "388bf2ac-33b8-40c2-a4c1-085dc435a3fa"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:30:28.829176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.829109    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle" pod "388bf2ac-33b8-40c2-a4c1-085dc435a3fa" (UID: "388bf2ac-33b8-40c2-a4c1-085dc435a3fa"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:28.829373 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.829354    4468 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-archives-path\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:28.829425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.829381    4468 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-service-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:28.829425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.829396    4468 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-trusted-ca-bundle\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:28.830926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.830902    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j" pod "388bf2ac-33b8-40c2-a4c1-085dc435a3fa" (UID: "388bf2ac-33b8-40c2-a4c1-085dc435a3fa"). InnerVolumeSpecName "kube-api-access-4kq6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:28.930481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:28.930435    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kq6j\" (UniqueName: \"kubernetes.io/projected/388bf2ac-33b8-40c2-a4c1-085dc435a3fa-kube-api-access-4kq6j\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:29.584520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:29.584487    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw"
Nov 01 06:30:29.584711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:29.584509    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cq9k6-jlkgw" event={"ID":"388bf2ac-33b8-40c2-a4c1-085dc435a3fa","Type":"ContainerDied","Data":"49d0e12e060b10bcabd30c873ca8466880aefacb1782aa3e493ce15c5e9174d0"}
Nov 01 06:30:29.584711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:29.584572    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d0e12e060b10bcabd30c873ca8466880aefacb1782aa3e493ce15c5e9174d0"
Nov 01 06:30:30.566397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.566363    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"]
Nov 01 06:30:30.567025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567010    4468 cpu_manager.go:414] "containerMap: removing container" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567027    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567036    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567041    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567057    4468 cpu_manager.go:414] "containerMap: removing container" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567064    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567074    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567080    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567086    4468 cpu_manager.go:414] "containerMap: removing container" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:30.567090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567090    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567100    4468 cpu_manager.go:414] "containerMap: removing container" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567104    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567114    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567118    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567132    4468 cpu_manager.go:414] "containerMap: removing container" podUID="388bf2ac-33b8-40c2-a4c1-085dc435a3fa" containerName="insights-gathering"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567137    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388bf2ac-33b8-40c2-a4c1-085dc435a3fa" containerName="insights-gathering"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567163    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567167    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567175    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567179    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567213    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567217    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567295    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567302    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567309    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567314    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567320    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567328    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567334    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567339    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:30.567339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567344    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:30.567929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567352    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:30.567929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567357    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:30.567929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.567362    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="388bf2ac-33b8-40c2-a4c1-085dc435a3fa" containerName="insights-gathering"
Nov 01 06:30:30.570978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.570963    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.626253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.626219    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"]
Nov 01 06:30:30.655346 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.655308    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.655580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.655550    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrzjw\" (UniqueName: \"kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.655636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.655612    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.670667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.670637    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:30:30.670856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.670759    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:30:30.670904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.670863    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:30:30.670959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.670943    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:30:30.756992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.756957    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.757267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.757222    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.757520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.757417    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrzjw\" (UniqueName: \"kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.757829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.757807    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.759857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.759839    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.813822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.813791    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrzjw\" (UniqueName: \"kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw\") pod \"maas-api-revoke-sub-36f71484-xs9s5\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:30.885136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:30.885051    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:31.262871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:31.262840    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"]
Nov 01 06:30:31.263442 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:31.263418    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893c890f_8d80_405f_a667_ffbbe0de5917.slice/crio-be6cb9049994ee1abec63985c1c9826e63e08e75b033bad1ca2029a55459fd42 WatchSource:0}: Error finding container be6cb9049994ee1abec63985c1c9826e63e08e75b033bad1ca2029a55459fd42: Status 404 returned error can't find the container with id be6cb9049994ee1abec63985c1c9826e63e08e75b033bad1ca2029a55459fd42
Nov 01 06:30:31.600510 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:31.600446    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5" event={"ID":"893c890f-8d80-405f-a667-ffbbe0de5917","Type":"ContainerStarted","Data":"f9cbc8fc5ee262269d5a7c8c7a6e7760ed01ee74648917ea09acfa27373f1907"}
Nov 01 06:30:31.600925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:31.600538    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5" event={"ID":"893c890f-8d80-405f-a667-ffbbe0de5917","Type":"ContainerStarted","Data":"be6cb9049994ee1abec63985c1c9826e63e08e75b033bad1ca2029a55459fd42"}
Nov 01 06:30:31.714958 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:31.714900    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5" podStartSLOduration=1.714886016 podStartE2EDuration="1.714886016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:31.711870323 +0000 UTC m=+1243.279677154" watchObservedRunningTime="2026-11-01 06:30:31.714886016 +0000 UTC m=+1243.282692845"
Nov 01 06:30:32.609611 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:32.609576    4468 generic.go:356] "Generic (PLEG): container finished" podID="893c890f-8d80-405f-a667-ffbbe0de5917" containerID="f9cbc8fc5ee262269d5a7c8c7a6e7760ed01ee74648917ea09acfa27373f1907" exitCode=0
Nov 01 06:30:32.610000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:32.609657    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5" event={"ID":"893c890f-8d80-405f-a667-ffbbe0de5917","Type":"ContainerDied","Data":"f9cbc8fc5ee262269d5a7c8c7a6e7760ed01ee74648917ea09acfa27373f1907"}
Nov 01 06:30:32.947956 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:30:32.947874    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:30:33.768544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.768517    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:33.902509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.902369    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca\") pod \"893c890f-8d80-405f-a667-ffbbe0de5917\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") "
Nov 01 06:30:33.902509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.902453    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca\") pod \"893c890f-8d80-405f-a667-ffbbe0de5917\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") "
Nov 01 06:30:33.902750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.902640    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw\" (UniqueName: \"kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw\") pod \"893c890f-8d80-405f-a667-ffbbe0de5917\" (UID: \"893c890f-8d80-405f-a667-ffbbe0de5917\") "
Nov 01 06:30:33.902878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.902854    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca" pod "893c890f-8d80-405f-a667-ffbbe0de5917" (UID: "893c890f-8d80-405f-a667-ffbbe0de5917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:33.903189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.903165    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:33.904951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.904916    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw" pod "893c890f-8d80-405f-a667-ffbbe0de5917" (UID: "893c890f-8d80-405f-a667-ffbbe0de5917"). InnerVolumeSpecName "kube-api-access-qrzjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:33.905186 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:33.905166    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca" pod "893c890f-8d80-405f-a667-ffbbe0de5917" (UID: "893c890f-8d80-405f-a667-ffbbe0de5917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:30:34.004307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.004268    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893c890f-8d80-405f-a667-ffbbe0de5917-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:34.004307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.004299    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrzjw\" (UniqueName: \"kubernetes.io/projected/893c890f-8d80-405f-a667-ffbbe0de5917-kube-api-access-qrzjw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:34.045634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.045599    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"]
Nov 01 06:30:34.050142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.050110    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70751ef-9g4df"]
Nov 01 06:30:34.590793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.590762    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c027ca87-e45e-4cd3-ab6a-8f45daca0472" path="/var/lib/kubelet/pods/c027ca87-e45e-4cd3-ab6a-8f45daca0472/volumes"
Nov 01 06:30:34.625977 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.625949    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"
Nov 01 06:30:34.626131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.625946    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5" event={"ID":"893c890f-8d80-405f-a667-ffbbe0de5917","Type":"ContainerDied","Data":"be6cb9049994ee1abec63985c1c9826e63e08e75b033bad1ca2029a55459fd42"}
Nov 01 06:30:34.626131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:34.626083    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be6cb9049994ee1abec63985c1c9826e63e08e75b033bad1ca2029a55459fd42"
Nov 01 06:30:37.882169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882134    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"]
Nov 01 06:30:37.882751 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882734    4468 cpu_manager.go:414] "containerMap: removing container" podUID="893c890f-8d80-405f-a667-ffbbe0de5917" containerName="revoke-keys"
Nov 01 06:30:37.882796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882752    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893c890f-8d80-405f-a667-ffbbe0de5917" containerName="revoke-keys"
Nov 01 06:30:37.882796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882768    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:37.882796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882774    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:37.882796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882789    4468 cpu_manager.go:414] "containerMap: removing container" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:37.882796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882794    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882804    4468 cpu_manager.go:414] "containerMap: removing container" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882809    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882818    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882822    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882845    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882849    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a35cd3f-145b-402a-9ee8-7aee929a0db4" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882923    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3b5262-03f3-434a-813b-3428689cfc98" containerName="cleanup"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882938    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="893c890f-8d80-405f-a667-ffbbe0de5917" containerName="revoke-keys"
Nov 01 06:30:37.882945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882946    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="975b1319-328e-492a-8303-087d150d5d7d" containerName="cleanup"
Nov 01 06:30:37.883201 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882951    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c374a-5646-47fd-bcfb-a92a71fda553" containerName="cleanup"
Nov 01 06:30:37.883201 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.882958    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="807be97f-c7b4-4e67-84a9-c61d4b291410" containerName="cleanup"
Nov 01 06:30:37.886519 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.886459    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:37.944498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.944451    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"]
Nov 01 06:30:37.958953 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.958912    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:37.959142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.959047    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpvf\" (UniqueName: \"kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:37.959246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.959224    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:37.989082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.989049    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:30:37.989261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.989165    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:30:37.989261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.989205    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:30:37.989261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:37.989205    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:30:38.060973 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.060941    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.061150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.060996    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpvf\" (UniqueName: \"kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.061150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.061062    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.061729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.061653    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.063416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.063393    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.117512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.117480    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpvf\" (UniqueName: \"kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf\") pod \"maas-api-revoke-sub-630e6309-g842h\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.200546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.200448    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:38.579249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.579218    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"]
Nov 01 06:30:38.580184 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:38.580156    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633be351_2af8_4cac_aaf9_42817db31ba2.slice/crio-f355094fa5965fae989d4410231bcfc1ee5290905e1feeaad84498b954b8b763 WatchSource:0}: Error finding container f355094fa5965fae989d4410231bcfc1ee5290905e1feeaad84498b954b8b763: Status 404 returned error can't find the container with id f355094fa5965fae989d4410231bcfc1ee5290905e1feeaad84498b954b8b763
Nov 01 06:30:38.698803 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:38.698768    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h" event={"ID":"633be351-2af8-4cac-aaf9-42817db31ba2","Type":"ContainerStarted","Data":"f355094fa5965fae989d4410231bcfc1ee5290905e1feeaad84498b954b8b763"}
Nov 01 06:30:39.708514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:39.708482    4468 generic.go:356] "Generic (PLEG): container finished" podID="633be351-2af8-4cac-aaf9-42817db31ba2" containerID="e1236503c9ab053d50cd90030977d52bce1f5ad812afaa25ddefde62931ea1b0" exitCode=0
Nov 01 06:30:39.708933 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:39.708570    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h" event={"ID":"633be351-2af8-4cac-aaf9-42817db31ba2","Type":"ContainerDied","Data":"e1236503c9ab053d50cd90030977d52bce1f5ad812afaa25ddefde62931ea1b0"}
Nov 01 06:30:40.861719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.861691    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:40.896356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.896321    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca\") pod \"633be351-2af8-4cac-aaf9-42817db31ba2\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") "
Nov 01 06:30:40.896570 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.896448    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca\") pod \"633be351-2af8-4cac-aaf9-42817db31ba2\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") "
Nov 01 06:30:40.896656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.896579    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf\" (UniqueName: \"kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf\") pod \"633be351-2af8-4cac-aaf9-42817db31ba2\" (UID: \"633be351-2af8-4cac-aaf9-42817db31ba2\") "
Nov 01 06:30:40.896715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.896677    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca" pod "633be351-2af8-4cac-aaf9-42817db31ba2" (UID: "633be351-2af8-4cac-aaf9-42817db31ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:40.896956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.896940    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:40.898745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.898723    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf" pod "633be351-2af8-4cac-aaf9-42817db31ba2" (UID: "633be351-2af8-4cac-aaf9-42817db31ba2"). InnerVolumeSpecName "kube-api-access-rvpvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:40.898745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.898733    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca" pod "633be351-2af8-4cac-aaf9-42817db31ba2" (UID: "633be351-2af8-4cac-aaf9-42817db31ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:30:40.998538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.998409    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpvf\" (UniqueName: \"kubernetes.io/projected/633be351-2af8-4cac-aaf9-42817db31ba2-kube-api-access-rvpvf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:40.998538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:40.998452    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633be351-2af8-4cac-aaf9-42817db31ba2-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:41.045173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:41.045135    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"]
Nov 01 06:30:41.049000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:41.048970    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ca83c8-l8rlv"]
Nov 01 06:30:41.724415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:41.724376    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h" event={"ID":"633be351-2af8-4cac-aaf9-42817db31ba2","Type":"ContainerDied","Data":"f355094fa5965fae989d4410231bcfc1ee5290905e1feeaad84498b954b8b763"}
Nov 01 06:30:41.724620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:41.724436    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f355094fa5965fae989d4410231bcfc1ee5290905e1feeaad84498b954b8b763"
Nov 01 06:30:41.724620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:41.724390    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"
Nov 01 06:30:42.600858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:42.600828    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89da4b7c-ac75-41f2-9873-324abdb5519d" path="/var/lib/kubelet/pods/89da4b7c-ac75-41f2-9873-324abdb5519d/volumes"
Nov 01 06:30:45.758159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.758130    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"]
Nov 01 06:30:45.758756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.758741    4468 cpu_manager.go:414] "containerMap: removing container" podUID="633be351-2af8-4cac-aaf9-42817db31ba2" containerName="revoke-keys"
Nov 01 06:30:45.758802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.758757    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633be351-2af8-4cac-aaf9-42817db31ba2" containerName="revoke-keys"
Nov 01 06:30:45.758899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.758889    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="633be351-2af8-4cac-aaf9-42817db31ba2" containerName="revoke-keys"
Nov 01 06:30:45.762713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.762695    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.815353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.815316    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:30:45.818625 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.818601    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"]
Nov 01 06:30:45.859340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.859291    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.859340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.859338    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbkh\" (UniqueName: \"kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.859647 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.859605    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.863847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.863826    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:30:45.863930 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.863829    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:30:45.864739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.864719    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:30:45.961109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.961071    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.961298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.961193    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.961298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.961223    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbkh\" (UniqueName: \"kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.961778 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.961752    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:45.963785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:45.963754    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:46.018065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:46.017994    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbkh\" (UniqueName: \"kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh\") pod \"maas-api-revoke-sub-af6a1470-kdtf8\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:46.076762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:46.076728    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:46.452944 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:46.452912    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb520194f_14ba_4651_95b3_5ba2b06444e3.slice/crio-bfdf58ef937a5bcdd1f6c47e848b685efc419725b2ea60c7a52b6cb2f717dc9a WatchSource:0}: Error finding container bfdf58ef937a5bcdd1f6c47e848b685efc419725b2ea60c7a52b6cb2f717dc9a: Status 404 returned error can't find the container with id bfdf58ef937a5bcdd1f6c47e848b685efc419725b2ea60c7a52b6cb2f717dc9a
Nov 01 06:30:46.453208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:46.453184    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"]
Nov 01 06:30:46.761541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:46.761505    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8" event={"ID":"b520194f-14ba-4651-95b3-5ba2b06444e3","Type":"ContainerStarted","Data":"2e54d98202f89e0d31838adfeb687c51a10596567b6125db9120c626c742daea"}
Nov 01 06:30:46.761969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:46.761569    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8" event={"ID":"b520194f-14ba-4651-95b3-5ba2b06444e3","Type":"ContainerStarted","Data":"bfdf58ef937a5bcdd1f6c47e848b685efc419725b2ea60c7a52b6cb2f717dc9a"}
Nov 01 06:30:46.875939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:46.875880    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8" podStartSLOduration=1.875864548 podStartE2EDuration="1.875864548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:46.873008895 +0000 UTC m=+1258.440815738" watchObservedRunningTime="2026-11-01 06:30:46.875864548 +0000 UTC m=+1258.443671377"
Nov 01 06:30:47.769946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:47.769855    4468 generic.go:356] "Generic (PLEG): container finished" podID="b520194f-14ba-4651-95b3-5ba2b06444e3" containerID="2e54d98202f89e0d31838adfeb687c51a10596567b6125db9120c626c742daea" exitCode=0
Nov 01 06:30:47.770323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:47.769941    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8" event={"ID":"b520194f-14ba-4651-95b3-5ba2b06444e3","Type":"ContainerDied","Data":"2e54d98202f89e0d31838adfeb687c51a10596567b6125db9120c626c742daea"}
Nov 01 06:30:48.933177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:48.933150    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:48.998425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:48.998392    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca\") pod \"b520194f-14ba-4651-95b3-5ba2b06444e3\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") "
Nov 01 06:30:48.998614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:48.998449    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca\") pod \"b520194f-14ba-4651-95b3-5ba2b06444e3\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") "
Nov 01 06:30:48.998614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:48.998527    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh\" (UniqueName: \"kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh\") pod \"b520194f-14ba-4651-95b3-5ba2b06444e3\" (UID: \"b520194f-14ba-4651-95b3-5ba2b06444e3\") "
Nov 01 06:30:48.998798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:48.998768    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca" pod "b520194f-14ba-4651-95b3-5ba2b06444e3" (UID: "b520194f-14ba-4651-95b3-5ba2b06444e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:48.999043 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:48.999028    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:49.000597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.000570    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca" pod "b520194f-14ba-4651-95b3-5ba2b06444e3" (UID: "b520194f-14ba-4651-95b3-5ba2b06444e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:30:49.000696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.000670    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh" pod "b520194f-14ba-4651-95b3-5ba2b06444e3" (UID: "b520194f-14ba-4651-95b3-5ba2b06444e3"). InnerVolumeSpecName "kube-api-access-qgbkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:49.099807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.099774    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbkh\" (UniqueName: \"kubernetes.io/projected/b520194f-14ba-4651-95b3-5ba2b06444e3-kube-api-access-qgbkh\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:49.099807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.099806    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520194f-14ba-4651-95b3-5ba2b06444e3-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:49.785830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.785793    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8" event={"ID":"b520194f-14ba-4651-95b3-5ba2b06444e3","Type":"ContainerDied","Data":"bfdf58ef937a5bcdd1f6c47e848b685efc419725b2ea60c7a52b6cb2f717dc9a"}
Nov 01 06:30:49.785830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.785815    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"
Nov 01 06:30:49.785830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:49.785839    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfdf58ef937a5bcdd1f6c47e848b685efc419725b2ea60c7a52b6cb2f717dc9a"
Nov 01 06:30:50.094723 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:50.094688    4468 scope.go:122] "RemoveContainer" containerID="dd6c239e792849b099df18f41c9abfab7cabf9b743864911fbb09f42165d0bb2"
Nov 01 06:30:50.104390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:50.104367    4468 scope.go:122] "RemoveContainer" containerID="550220841f074168575ab7bf0ab606829e88da232ba20995aaeaa00368aac9c1"
Nov 01 06:30:50.114091 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:50.114064    4468 scope.go:122] "RemoveContainer" containerID="71914a5909c51f41f05a642d09efd89bc4a1ab779d14803dc0a6fb07f8b1a6d7"
Nov 01 06:30:50.926078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:50.926041    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:30:53.347535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:53.347488    4468 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:30:53.590497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:53.589577    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:30:53.590497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:53.589653    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:30:53.724557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:53.724321    4468 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:30:53.724557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:53.724379    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:30:53.743621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:53.743574    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:30:54.749485 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.749432    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"]
Nov 01 06:30:54.750663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.750627    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b520194f-14ba-4651-95b3-5ba2b06444e3" containerName="revoke-keys"
Nov 01 06:30:54.750663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.750655    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b520194f-14ba-4651-95b3-5ba2b06444e3" containerName="revoke-keys"
Nov 01 06:30:54.750996 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.750800    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b520194f-14ba-4651-95b3-5ba2b06444e3" containerName="revoke-keys"
Nov 01 06:30:54.757077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.757041    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.810692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.810660    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"]
Nov 01 06:30:54.860490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.858936    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:30:54.860490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.859061    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:30:54.860490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.859143    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:30:54.860490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.859190    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:30:54.888489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.887530    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.888489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.887793    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk78l\" (UniqueName: \"kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.888489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.887842    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.989585 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.989540    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.989847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.989828    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk78l\" (UniqueName: \"kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.989947 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.989876    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.990749 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.990721    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:54.992844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:54.992822    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:55.047672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:55.047640    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk78l\" (UniqueName: \"kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l\") pod \"maas-api-revoke-sub-937897ad-w26fh\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:55.075334 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:55.075297    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:55.451583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:55.451555    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"]
Nov 01 06:30:55.452309 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:30:55.452286    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135756d4_93a2_465e_b896_03c577188d78.slice/crio-b7377e0451522cbfd91927fa98fb6d6fd3b2edd3d6f723634313ffa2bd750b56 WatchSource:0}: Error finding container b7377e0451522cbfd91927fa98fb6d6fd3b2edd3d6f723634313ffa2bd750b56: Status 404 returned error can't find the container with id b7377e0451522cbfd91927fa98fb6d6fd3b2edd3d6f723634313ffa2bd750b56
Nov 01 06:30:55.962374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:55.962280    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh" event={"ID":"135756d4-93a2-465e-b896-03c577188d78","Type":"ContainerStarted","Data":"3152f0e9e9344b82d259c093e60b2fa6189c9331cbd2c48e7fbb6507378a72d5"}
Nov 01 06:30:55.962374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:55.962351    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh" event={"ID":"135756d4-93a2-465e-b896-03c577188d78","Type":"ContainerStarted","Data":"b7377e0451522cbfd91927fa98fb6d6fd3b2edd3d6f723634313ffa2bd750b56"}
Nov 01 06:30:56.076779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:56.076713    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh" podStartSLOduration=2.076696374 podStartE2EDuration="2.076696374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:56.074515645 +0000 UTC m=+1267.642322477" watchObservedRunningTime="2026-11-01 06:30:56.076696374 +0000 UTC m=+1267.644503205"
Nov 01 06:30:56.969855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:56.969823    4468 generic.go:356] "Generic (PLEG): container finished" podID="135756d4-93a2-465e-b896-03c577188d78" containerID="3152f0e9e9344b82d259c093e60b2fa6189c9331cbd2c48e7fbb6507378a72d5" exitCode=0
Nov 01 06:30:56.970233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:56.969919    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh" event={"ID":"135756d4-93a2-465e-b896-03c577188d78","Type":"ContainerDied","Data":"3152f0e9e9344b82d259c093e60b2fa6189c9331cbd2c48e7fbb6507378a72d5"}
Nov 01 06:30:58.137190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.137165    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:58.233144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.233057    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca\") pod \"135756d4-93a2-465e-b896-03c577188d78\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") "
Nov 01 06:30:58.233144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.233093    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca\") pod \"135756d4-93a2-465e-b896-03c577188d78\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") "
Nov 01 06:30:58.233397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.233217    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l\" (UniqueName: \"kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l\") pod \"135756d4-93a2-465e-b896-03c577188d78\" (UID: \"135756d4-93a2-465e-b896-03c577188d78\") "
Nov 01 06:30:58.233493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.233403    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca" pod "135756d4-93a2-465e-b896-03c577188d78" (UID: "135756d4-93a2-465e-b896-03c577188d78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:58.233554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.233531    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135756d4-93a2-465e-b896-03c577188d78-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:58.235447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.235420    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca" pod "135756d4-93a2-465e-b896-03c577188d78" (UID: "135756d4-93a2-465e-b896-03c577188d78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:30:58.235619 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.235601    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l" pod "135756d4-93a2-465e-b896-03c577188d78" (UID: "135756d4-93a2-465e-b896-03c577188d78"). InnerVolumeSpecName "kube-api-access-dk78l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:58.334480 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.334426    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135756d4-93a2-465e-b896-03c577188d78-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:58.334480 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.334454    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk78l\" (UniqueName: \"kubernetes.io/projected/135756d4-93a2-465e-b896-03c577188d78-kube-api-access-dk78l\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:58.985278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.985193    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"
Nov 01 06:30:58.985278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.985230    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh" event={"ID":"135756d4-93a2-465e-b896-03c577188d78","Type":"ContainerDied","Data":"b7377e0451522cbfd91927fa98fb6d6fd3b2edd3d6f723634313ffa2bd750b56"}
Nov 01 06:30:58.985523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:30:58.985287    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7377e0451522cbfd91927fa98fb6d6fd3b2edd3d6f723634313ffa2bd750b56"
Nov 01 06:31:01.051086 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:01.051054    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"]
Nov 01 06:31:01.100597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:01.100566    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb4b916-lf8w6"]
Nov 01 06:31:02.591148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:02.591112    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853d32de-15c6-497d-92ab-646db5f878cd" path="/var/lib/kubelet/pods/853d32de-15c6-497d-92ab-646db5f878cd/volumes"
Nov 01 06:31:02.948103 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:31:02.948006    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:31:09.904750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.904721    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"]
Nov 01 06:31:09.905276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.905262    4468 cpu_manager.go:414] "containerMap: removing container" podUID="135756d4-93a2-465e-b896-03c577188d78" containerName="revoke-keys"
Nov 01 06:31:09.905329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.905277    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135756d4-93a2-465e-b896-03c577188d78" containerName="revoke-keys"
Nov 01 06:31:09.905404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.905395    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="135756d4-93a2-465e-b896-03c577188d78" containerName="revoke-keys"
Nov 01 06:31:09.908903 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.908888    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:09.965335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.965300    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"]
Nov 01 06:31:09.985054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.985022    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:09.985217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.985060    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jqf\" (UniqueName: \"kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:09.985217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:09.985199    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.010678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.010651    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:31:10.010846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.010701    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:31:10.010846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.010714    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:31:10.010846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.010773    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:31:10.086720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.086687    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.086720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.086720    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jqf\" (UniqueName: \"kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.086920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.086790    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.087411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.087374    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.089207 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.089183    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.143167 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.143129    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jqf\" (UniqueName: \"kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf\") pod \"maas-api-revoke-sub-75e8fe5a-pbqnn\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.222155 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.222073    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:10.597633 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:31:10.597599    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d367d20_645e_4745_b704_194e688ce8df.slice/crio-99f7bc8fb7422a83604b522d543a9a16cf07a147dc0883552d232e45f36c3fff WatchSource:0}: Error finding container 99f7bc8fb7422a83604b522d543a9a16cf07a147dc0883552d232e45f36c3fff: Status 404 returned error can't find the container with id 99f7bc8fb7422a83604b522d543a9a16cf07a147dc0883552d232e45f36c3fff
Nov 01 06:31:10.597794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:10.597770    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"]
Nov 01 06:31:11.128041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:11.127990    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn" event={"ID":"3d367d20-645e-4745-b704-194e688ce8df","Type":"ContainerStarted","Data":"029dd20bee9d86ce338ac75c8d4ec69cdb2606247656a3c54de687cc85d0288e"}
Nov 01 06:31:11.128671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:11.128061    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn" event={"ID":"3d367d20-645e-4745-b704-194e688ce8df","Type":"ContainerStarted","Data":"99f7bc8fb7422a83604b522d543a9a16cf07a147dc0883552d232e45f36c3fff"}
Nov 01 06:31:11.248586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:11.248436    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn" podStartSLOduration=2.248415542 podStartE2EDuration="2.248415542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:11.242901693 +0000 UTC m=+1282.810708525" watchObservedRunningTime="2026-11-01 06:31:11.248415542 +0000 UTC m=+1282.816222372"
Nov 01 06:31:12.062530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:12.062491    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"]
Nov 01 06:31:12.068136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:12.068099    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e980b43b-84cb8"]
Nov 01 06:31:12.136244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:12.136213    4468 generic.go:356] "Generic (PLEG): container finished" podID="3d367d20-645e-4745-b704-194e688ce8df" containerID="029dd20bee9d86ce338ac75c8d4ec69cdb2606247656a3c54de687cc85d0288e" exitCode=0
Nov 01 06:31:12.136732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:12.136297    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn" event={"ID":"3d367d20-645e-4745-b704-194e688ce8df","Type":"ContainerDied","Data":"029dd20bee9d86ce338ac75c8d4ec69cdb2606247656a3c54de687cc85d0288e"}
Nov 01 06:31:12.591288 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:12.591255    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f2bcee-8b32-4719-8b35-b1cab94d1f07" path="/var/lib/kubelet/pods/09f2bcee-8b32-4719-8b35-b1cab94d1f07/volumes"
Nov 01 06:31:13.302951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.302921    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:13.440583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.440448    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca\") pod \"3d367d20-645e-4745-b704-194e688ce8df\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") "
Nov 01 06:31:13.440583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.440578    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf\" (UniqueName: \"kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf\") pod \"3d367d20-645e-4745-b704-194e688ce8df\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") "
Nov 01 06:31:13.440822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.440599    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca\") pod \"3d367d20-645e-4745-b704-194e688ce8df\" (UID: \"3d367d20-645e-4745-b704-194e688ce8df\") "
Nov 01 06:31:13.440885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.440831    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca" pod "3d367d20-645e-4745-b704-194e688ce8df" (UID: "3d367d20-645e-4745-b704-194e688ce8df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:13.441115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.441092    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d367d20-645e-4745-b704-194e688ce8df-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:13.443384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.443361    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf" pod "3d367d20-645e-4745-b704-194e688ce8df" (UID: "3d367d20-645e-4745-b704-194e688ce8df"). InnerVolumeSpecName "kube-api-access-49jqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:13.443521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.443442    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca" pod "3d367d20-645e-4745-b704-194e688ce8df" (UID: "3d367d20-645e-4745-b704-194e688ce8df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:13.541937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.541902    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jqf\" (UniqueName: \"kubernetes.io/projected/3d367d20-645e-4745-b704-194e688ce8df-kube-api-access-49jqf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:13.541937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:13.541934    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d367d20-645e-4745-b704-194e688ce8df-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:14.152445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:14.152411    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn" event={"ID":"3d367d20-645e-4745-b704-194e688ce8df","Type":"ContainerDied","Data":"99f7bc8fb7422a83604b522d543a9a16cf07a147dc0883552d232e45f36c3fff"}
Nov 01 06:31:14.152629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:14.152461    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f7bc8fb7422a83604b522d543a9a16cf07a147dc0883552d232e45f36c3fff"
Nov 01 06:31:14.152629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:14.152426    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"
Nov 01 06:31:19.120355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:19.120315    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"]
Nov 01 06:31:19.120893 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:19.120595    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy" containerID="cri-o://8ad0a1f14db500d811011ff4f2f4c7b4b908d55a6273e4caf1d4be747640de1f" gracePeriod=30
Nov 01 06:31:20.533294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.533257    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:31:20.658862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.658827    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"]
Nov 01 06:31:20.659590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.659568    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3d367d20-645e-4745-b704-194e688ce8df" containerName="revoke-keys"
Nov 01 06:31:20.659590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.659590    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d367d20-645e-4745-b704-194e688ce8df" containerName="revoke-keys"
Nov 01 06:31:20.659794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.659779    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d367d20-645e-4745-b704-194e688ce8df" containerName="revoke-keys"
Nov 01 06:31:20.663566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.663546    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.716541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.716510    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:31:20.716541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.716538    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:31:20.716804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.716598    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:31:20.721373 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.721343    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"]
Nov 01 06:31:20.765205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.765179    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:31:20.855490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.855256    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfr44\" (UniqueName: \"kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.855490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.855420    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.855490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.855483    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.956357 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.956315    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfr44\" (UniqueName: \"kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.956612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.956502    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.956612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.956552    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.957146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.957117    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:20.959078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:20.959061    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:21.012672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:21.012646    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfr44\" (UniqueName: \"kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44\") pod \"maas-api-revoke-sub-d0394fa4-q5gt4\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:21.277127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:21.277052    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:21.684489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:21.682158    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"]
Nov 01 06:31:22.224075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:22.223991    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4" event={"ID":"6900cf21-eafa-436b-85c0-fdca5918d958","Type":"ContainerStarted","Data":"e2e918b3e2fd4730fa8376626784f3e2ba448b47265a6b478df7bfa971e091e0"}
Nov 01 06:31:23.238991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:23.238955    4468 generic.go:356] "Generic (PLEG): container finished" podID="6900cf21-eafa-436b-85c0-fdca5918d958" containerID="8a0ff1becc4fca14542fb32f0f332fec36fc6209c58c5901f8e3f552609aaf1c" exitCode=0
Nov 01 06:31:23.239497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:23.239219    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4" event={"ID":"6900cf21-eafa-436b-85c0-fdca5918d958","Type":"ContainerDied","Data":"8a0ff1becc4fca14542fb32f0f332fec36fc6209c58c5901f8e3f552609aaf1c"}
Nov 01 06:31:24.248968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.248927    4468 generic.go:356] "Generic (PLEG): container finished" podID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerID="8ad0a1f14db500d811011ff4f2f4c7b4b908d55a6273e4caf1d4be747640de1f" exitCode=0
Nov 01 06:31:24.249357 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.248998    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" event={"ID":"db5ba475-fdb0-417a-8799-d8448ad561dc","Type":"ContainerDied","Data":"8ad0a1f14db500d811011ff4f2f4c7b4b908d55a6273e4caf1d4be747640de1f"}
Nov 01 06:31:24.463689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.463663    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:31:24.465752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.465735    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.576854    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44\" (UniqueName: \"kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44\") pod \"6900cf21-eafa-436b-85c0-fdca5918d958\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.576915    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.576958    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577010    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577040    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577079    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577188    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577282    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577311    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca\") pod \"6900cf21-eafa-436b-85c0-fdca5918d958\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") "
Nov 01 06:31:24.577429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577354    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.578035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577453    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca\") pod \"6900cf21-eafa-436b-85c0-fdca5918d958\" (UID: \"6900cf21-eafa-436b-85c0-fdca5918d958\") "
Nov 01 06:31:24.578035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577532    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy\") pod \"db5ba475-fdb0-417a-8799-d8448ad561dc\" (UID: \"db5ba475-fdb0-417a-8799-d8448ad561dc\") "
Nov 01 06:31:24.578035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:31:24.578035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577742    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:24.578035 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.577932    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:31:24.578279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578053    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:31:24.578279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578238    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca" pod "6900cf21-eafa-436b-85c0-fdca5918d958" (UID: "6900cf21-eafa-436b-85c0-fdca5918d958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:24.578524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578401    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db5ba475-fdb0-417a-8799-d8448ad561dc-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.578524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578424    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.578524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578438    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.578524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578479    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.578524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.578496    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.580620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.580593    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca" pod "6900cf21-eafa-436b-85c0-fdca5918d958" (UID: "6900cf21-eafa-436b-85c0-fdca5918d958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:24.581750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.581714    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:31:24.583819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.583793    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:31:24.583939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.583830    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:24.583939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.583875    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:31:24.583939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.583917    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44" pod "6900cf21-eafa-436b-85c0-fdca5918d958" (UID: "6900cf21-eafa-436b-85c0-fdca5918d958"). InnerVolumeSpecName "kube-api-access-pfr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:24.585930 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.585898    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l" pod "db5ba475-fdb0-417a-8799-d8448ad561dc" (UID: "db5ba475-fdb0-417a-8799-d8448ad561dc"). InnerVolumeSpecName "kube-api-access-9729l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:24.680591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680512    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6900cf21-eafa-436b-85c0-fdca5918d958-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.680591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680551    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.680591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680568    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfr44\" (UniqueName: \"kubernetes.io/projected/6900cf21-eafa-436b-85c0-fdca5918d958-kube-api-access-pfr44\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.680591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680585    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db5ba475-fdb0-417a-8799-d8448ad561dc-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.680836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680598    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.680836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680612    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db5ba475-fdb0-417a-8799-d8448ad561dc-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.680836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:24.680626    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9729l\" (UniqueName: \"kubernetes.io/projected/db5ba475-fdb0-417a-8799-d8448ad561dc-kube-api-access-9729l\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:25.257908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.257879    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"
Nov 01 06:31:25.257908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.257888    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4" event={"ID":"6900cf21-eafa-436b-85c0-fdca5918d958","Type":"ContainerDied","Data":"e2e918b3e2fd4730fa8376626784f3e2ba448b47265a6b478df7bfa971e091e0"}
Nov 01 06:31:25.258389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.257971    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e918b3e2fd4730fa8376626784f3e2ba448b47265a6b478df7bfa971e091e0"
Nov 01 06:31:25.259487 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.259444    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"
Nov 01 06:31:25.259487 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.259455    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg" event={"ID":"db5ba475-fdb0-417a-8799-d8448ad561dc","Type":"ContainerDied","Data":"945628bcd9370a10571911985860c8d8865af167a72fff07281f4a16fdaa09a6"}
Nov 01 06:31:25.259637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.259534    4468 scope.go:122] "RemoveContainer" containerID="8ad0a1f14db500d811011ff4f2f4c7b4b908d55a6273e4caf1d4be747640de1f"
Nov 01 06:31:25.476723 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.476692    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"]
Nov 01 06:31:25.526033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:25.525941    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-6zskg"]
Nov 01 06:31:26.590595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:26.590568    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" path="/var/lib/kubelet/pods/db5ba475-fdb0-417a-8799-d8448ad561dc/volumes"
Nov 01 06:31:27.709616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:27.709583    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl_ca1817c8-1355-456e-a943-5f5d045df3c6/payload-processing/0.log"
Nov 01 06:31:28.045362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:28.045337    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"]
Nov 01 06:31:28.094700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:28.094650    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3231be4-2xh2s"]
Nov 01 06:31:28.483831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:28.483759    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x_0a783741-283e-4888-9ec0-2d5aa66d8607/payload-processing/0.log"
Nov 01 06:31:28.591037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:28.591002    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a65f9e-aaab-4d0b-9253-59a7f4108227" path="/var/lib/kubelet/pods/84a65f9e-aaab-4d0b-9253-59a7f4108227/volumes"
Nov 01 06:31:32.948155 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:31:32.948074    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:31:38.046374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.046339    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"]
Nov 01 06:31:38.095568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.095526    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13c2146-kmg2m"]
Nov 01 06:31:38.590779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.590745    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41a81b9-6f64-42b5-afab-03e1eabcedb3" path="/var/lib/kubelet/pods/e41a81b9-6f64-42b5-afab-03e1eabcedb3/volumes"
Nov 01 06:31:38.727367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.727333    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"]
Nov 01 06:31:38.727891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.727878    4468 cpu_manager.go:414] "containerMap: removing container" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy"
Nov 01 06:31:38.727951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.727893    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy"
Nov 01 06:31:38.727951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.727932    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6900cf21-eafa-436b-85c0-fdca5918d958" containerName="revoke-keys"
Nov 01 06:31:38.727951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.727937    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6900cf21-eafa-436b-85c0-fdca5918d958" containerName="revoke-keys"
Nov 01 06:31:38.728051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.728041    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6900cf21-eafa-436b-85c0-fdca5918d958" containerName="revoke-keys"
Nov 01 06:31:38.728090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.728057    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5ba475-fdb0-417a-8799-d8448ad561dc" containerName="istio-proxy"
Nov 01 06:31:38.731622 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.731606    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.758881 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.758854    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.758971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.758900    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.759039 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.759024    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ss5\" (UniqueName: \"kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.788249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.788224    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"]
Nov 01 06:31:38.833598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.833571    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:31:38.833743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.833641    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:31:38.833743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.833664    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:31:38.833743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.833717    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:31:38.860378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.860318    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.860378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.860364    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.860542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.860412    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ss5\" (UniqueName: \"kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.860944 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.860916    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.862855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.862835    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:38.916338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:38.916312    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ss5\" (UniqueName: \"kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5\") pod \"maas-api-revoke-sub-4ab73428-nn9l5\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:39.044685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:39.044656    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:39.418742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:39.418717    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"]
Nov 01 06:31:39.419354 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:31:39.419330    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96d328a_eefe_4560_987c_dbb75f304a31.slice/crio-e0971b68619c4182e1732eeb9409f12b71990476bfa3882e4ddf8fcb80dc2941 WatchSource:0}: Error finding container e0971b68619c4182e1732eeb9409f12b71990476bfa3882e4ddf8fcb80dc2941: Status 404 returned error can't find the container with id e0971b68619c4182e1732eeb9409f12b71990476bfa3882e4ddf8fcb80dc2941
Nov 01 06:31:40.390411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:40.390375    4468 generic.go:356] "Generic (PLEG): container finished" podID="d96d328a-eefe-4560-987c-dbb75f304a31" containerID="bcc31e710ba238282b924c0b3877e931fe43224814678f557572ea48cc49652f" exitCode=0
Nov 01 06:31:40.390632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:40.390441    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5" event={"ID":"d96d328a-eefe-4560-987c-dbb75f304a31","Type":"ContainerDied","Data":"bcc31e710ba238282b924c0b3877e931fe43224814678f557572ea48cc49652f"}
Nov 01 06:31:40.390632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:40.390540    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5" event={"ID":"d96d328a-eefe-4560-987c-dbb75f304a31","Type":"ContainerStarted","Data":"e0971b68619c4182e1732eeb9409f12b71990476bfa3882e4ddf8fcb80dc2941"}
Nov 01 06:31:41.564029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.564000    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:41.594905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.594869    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca\") pod \"d96d328a-eefe-4560-987c-dbb75f304a31\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") "
Nov 01 06:31:41.595102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.594919    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca\") pod \"d96d328a-eefe-4560-987c-dbb75f304a31\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") "
Nov 01 06:31:41.595102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.595016    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5\" (UniqueName: \"kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5\") pod \"d96d328a-eefe-4560-987c-dbb75f304a31\" (UID: \"d96d328a-eefe-4560-987c-dbb75f304a31\") "
Nov 01 06:31:41.595244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.595219    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca" pod "d96d328a-eefe-4560-987c-dbb75f304a31" (UID: "d96d328a-eefe-4560-987c-dbb75f304a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:41.595447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.595431    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:41.597201 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.597165    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca" pod "d96d328a-eefe-4560-987c-dbb75f304a31" (UID: "d96d328a-eefe-4560-987c-dbb75f304a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:41.597310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.597283    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5" pod "d96d328a-eefe-4560-987c-dbb75f304a31" (UID: "d96d328a-eefe-4560-987c-dbb75f304a31"). InnerVolumeSpecName "kube-api-access-55ss5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:41.696455 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.696365    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ss5\" (UniqueName: \"kubernetes.io/projected/d96d328a-eefe-4560-987c-dbb75f304a31-kube-api-access-55ss5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:41.696455 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:41.696395    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96d328a-eefe-4560-987c-dbb75f304a31-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:42.405375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:42.405345    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"
Nov 01 06:31:42.405375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:42.405359    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5" event={"ID":"d96d328a-eefe-4560-987c-dbb75f304a31","Type":"ContainerDied","Data":"e0971b68619c4182e1732eeb9409f12b71990476bfa3882e4ddf8fcb80dc2941"}
Nov 01 06:31:42.405614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:42.405401    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0971b68619c4182e1732eeb9409f12b71990476bfa3882e4ddf8fcb80dc2941"
Nov 01 06:31:43.045407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.045371    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"]
Nov 01 06:31:43.094379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.094339    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6e463f-m88lt"]
Nov 01 06:31:43.373130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.373048    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"]
Nov 01 06:31:43.373628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.373605    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d96d328a-eefe-4560-987c-dbb75f304a31" containerName="revoke-keys"
Nov 01 06:31:43.373676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.373629    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96d328a-eefe-4560-987c-dbb75f304a31" containerName="revoke-keys"
Nov 01 06:31:43.373750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.373741    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96d328a-eefe-4560-987c-dbb75f304a31" containerName="revoke-keys"
Nov 01 06:31:43.377317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.377298    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.395169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.395148    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:31:43.418991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.418955    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.419147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.419043    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjsh\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.419147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.419066    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.419147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.419091    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.419254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.419184    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.419254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.419218    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.419391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.419369    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.431198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.431179    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 06:31:43.431307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.431180    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 06:31:43.431307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.431195    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 06:31:43.431373 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.431325    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7btt7\""
Nov 01 06:31:43.478594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.478566    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"]
Nov 01 06:31:43.521163 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521116    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.521359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521172    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjsh\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.521359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521205    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.521359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521229    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.521359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521268    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.521652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521539    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.521711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.521649    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.522409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.522383    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.523859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.523814    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.524194 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.524174    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.524286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.524226    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.524531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.524513    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.625298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.625220    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.625298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.625226    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjsh\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7glh\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:43.689870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:43.689834    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:44.160352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:44.160320    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"]
Nov 01 06:31:44.161220 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:31:44.161194    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288cfc7d_4091_4bef_949b_5b6ee8900f8b.slice/crio-114d22f0363fb162ec50747ae38126525c68a425c3c71732ba2eebf2fee67693 WatchSource:0}: Error finding container 114d22f0363fb162ec50747ae38126525c68a425c3c71732ba2eebf2fee67693: Status 404 returned error can't find the container with id 114d22f0363fb162ec50747ae38126525c68a425c3c71732ba2eebf2fee67693
Nov 01 06:31:44.422200 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:44.422112    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" event={"ID":"288cfc7d-4091-4bef-949b-5b6ee8900f8b","Type":"ContainerStarted","Data":"114d22f0363fb162ec50747ae38126525c68a425c3c71732ba2eebf2fee67693"}
Nov 01 06:31:44.621288 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:44.610662    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369" path="/var/lib/kubelet/pods/03ae4d0e-f57a-4aaa-8c6d-7e3c7c65d369/volumes"
Nov 01 06:31:47.975575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:47.975542    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:31:47.975856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:47.975607    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:31:48.501907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:48.501864    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" event={"ID":"288cfc7d-4091-4bef-949b-5b6ee8900f8b","Type":"ContainerStarted","Data":"b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27"}
Nov 01 06:31:48.502122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:48.501957    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:48.716987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:48.716937    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" podStartSLOduration=1.906416117 podStartE2EDuration="5.716918789s" totalImagesPullingTime="3.810502672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:48.666270419 +0000 UTC m=+1320.234077273" watchObservedRunningTime="2026-11-01 06:31:48.716918789 +0000 UTC m=+1320.284725620"
Nov 01 06:31:49.514609 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:49.514579    4468 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j7glh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:31:49.515104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:49.515064    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" podUID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:31:50.176570 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:50.176541    4468 scope.go:122] "RemoveContainer" containerID="00b41ca9a3389cfd607b5f8e28a067b9f2576c22fc23e8cd3ca4494ac66f91a6"
Nov 01 06:31:50.186912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:50.186894    4468 scope.go:122] "RemoveContainer" containerID="2afbb1b6572b872d89010f24570b2618b2064ab99d80a2d82c49a08351a6c900"
Nov 01 06:31:50.198599 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:50.198581    4468 scope.go:122] "RemoveContainer" containerID="5dc5fb0a16c55d10f112c249ad83c5b245cb09ebf235d3e16c659c1777e49f84"
Nov 01 06:31:50.207099 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:50.207083    4468 scope.go:122] "RemoveContainer" containerID="1eb8e233287d534872d9ca8f3d1688c61824cdebe2722383bb6f20d5a6ea435a"
Nov 01 06:31:50.215959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:50.215932    4468 scope.go:122] "RemoveContainer" containerID="d9b80fd818b55f6de4c86af56cfa37b33a50eb2dfe2e6ebf71b9298d95a7490c"
Nov 01 06:31:50.224628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:50.224609    4468 scope.go:122] "RemoveContainer" containerID="4590fe010d17d67565103cb4240b594117970127bcf06949049562471e607891"
Nov 01 06:31:52.514113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:52.514082    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:31:53.046544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:53.046501    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"]
Nov 01 06:31:53.049749 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:53.049726    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-026954d6-txf9d"]
Nov 01 06:31:54.590866 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:31:54.590829    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073364d1-fdbf-4f91-aae0-e90cfdd1f256" path="/var/lib/kubelet/pods/073364d1-fdbf-4f91-aae0-e90cfdd1f256/volumes"
Nov 01 06:32:02.947943 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:32:02.947900    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:32:11.453966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.453928    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"]
Nov 01 06:32:11.458655 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.458629    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.535653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.535613    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"]
Nov 01 06:32:11.559300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.559269    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:32:11.560083 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.560053    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:32:11.560475 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.560352    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:32:11.560475 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.560435    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:32:11.566417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.566381    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.566598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.566502    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.566757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.566729    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6rs\" (UniqueName: \"kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.667850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.667803    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.668062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.667963    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.668131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.668113    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6rs\" (UniqueName: \"kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.668514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.668491    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.670877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.670854    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.724853 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.724778    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6rs\" (UniqueName: \"kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs\") pod \"maas-api-revoke-sub-cb43ac68-w2nlf\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:11.773136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:11.773093    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:12.150814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:12.150783    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"]
Nov 01 06:32:12.151058 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:32:12.151032    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5145af_8c4f_41f2_9e58_e93f3a0fc12d.slice/crio-b851d22bd96ef54a056cc4853817a6dc3cda1d540a953c4007e4f4bd84342dae WatchSource:0}: Error finding container b851d22bd96ef54a056cc4853817a6dc3cda1d540a953c4007e4f4bd84342dae: Status 404 returned error can't find the container with id b851d22bd96ef54a056cc4853817a6dc3cda1d540a953c4007e4f4bd84342dae
Nov 01 06:32:12.707829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:12.707738    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf" event={"ID":"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d","Type":"ContainerStarted","Data":"a11dae01161079fc8a28eb67f6b7c22ac442c6f588fcce120f7a2f0b0164c296"}
Nov 01 06:32:12.707829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:12.707801    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf" event={"ID":"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d","Type":"ContainerStarted","Data":"b851d22bd96ef54a056cc4853817a6dc3cda1d540a953c4007e4f4bd84342dae"}
Nov 01 06:32:12.823296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:12.823243    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf" podStartSLOduration=1.82322801 podStartE2EDuration="1.82322801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:12.819802088 +0000 UTC m=+1344.387608956" watchObservedRunningTime="2026-11-01 06:32:12.82322801 +0000 UTC m=+1344.391034842"
Nov 01 06:32:13.716243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:13.716205    4468 generic.go:356] "Generic (PLEG): container finished" podID="0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" containerID="a11dae01161079fc8a28eb67f6b7c22ac442c6f588fcce120f7a2f0b0164c296" exitCode=0
Nov 01 06:32:13.716694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:13.716300    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf" event={"ID":"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d","Type":"ContainerDied","Data":"a11dae01161079fc8a28eb67f6b7c22ac442c6f588fcce120f7a2f0b0164c296"}
Nov 01 06:32:14.970643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:14.970146    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:15.032984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.032895    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs\" (UniqueName: \"kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs\") pod \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") "
Nov 01 06:32:15.032984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.032961    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca\") pod \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") "
Nov 01 06:32:15.033179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.033122    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca\") pod \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\" (UID: \"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d\") "
Nov 01 06:32:15.033304 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.033276    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca" pod "0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" (UID: "0e5145af-8c4f-41f2-9e58-e93f3a0fc12d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:32:15.033733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.033638    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:15.035308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.035277    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca" pod "0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" (UID: "0e5145af-8c4f-41f2-9e58-e93f3a0fc12d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:32:15.035452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.035434    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs" pod "0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" (UID: "0e5145af-8c4f-41f2-9e58-e93f3a0fc12d"). InnerVolumeSpecName "kube-api-access-nq6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:32:15.135142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.135110    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:15.135142 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.135140    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq6rs\" (UniqueName: \"kubernetes.io/projected/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d-kube-api-access-nq6rs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:15.197591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.197555    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:32:15.732516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.732450    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"
Nov 01 06:32:15.732516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.732508    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf" event={"ID":"0e5145af-8c4f-41f2-9e58-e93f3a0fc12d","Type":"ContainerDied","Data":"b851d22bd96ef54a056cc4853817a6dc3cda1d540a953c4007e4f4bd84342dae"}
Nov 01 06:32:15.732746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:15.732558    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b851d22bd96ef54a056cc4853817a6dc3cda1d540a953c4007e4f4bd84342dae"
Nov 01 06:32:32.947922 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:32:32.947883    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:32:39.232781 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:39.232750    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:32:41.046394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:41.046356    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"]
Nov 01 06:32:41.095855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:41.095820    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eecc25-npsfs"]
Nov 01 06:32:42.590541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:42.590510    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f62809-f845-43c3-84b6-5bf1bfe7aac1" path="/var/lib/kubelet/pods/35f62809-f845-43c3-84b6-5bf1bfe7aac1/volumes"
Nov 01 06:32:49.350050 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.350012    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"]
Nov 01 06:32:49.350847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.350801    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl" podUID="ca1817c8-1355-456e-a943-5f5d045df3c6" containerName="payload-processing" containerID="cri-o://b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330" gracePeriod=30
Nov 01 06:32:49.403764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.403214    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"]
Nov 01 06:32:49.403764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.403606    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9" podUID="a5d94d75-dde2-42b5-9aee-cfdba3875810" containerName="payload-pre-processing" containerID="cri-o://76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578" gracePeriod=30
Nov 01 06:32:49.693972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.693941    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:32:49.714006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.713984    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:32:49.804065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.804036    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk\" (UniqueName: \"kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk\") pod \"a5d94d75-dde2-42b5-9aee-cfdba3875810\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") "
Nov 01 06:32:49.804260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.804093    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume\") pod \"a5d94d75-dde2-42b5-9aee-cfdba3875810\" (UID: \"a5d94d75-dde2-42b5-9aee-cfdba3875810\") "
Nov 01 06:32:49.804529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.804500    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume" pod "a5d94d75-dde2-42b5-9aee-cfdba3875810" (UID: "a5d94d75-dde2-42b5-9aee-cfdba3875810"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:32:49.807028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.806998    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk" pod "a5d94d75-dde2-42b5-9aee-cfdba3875810" (UID: "a5d94d75-dde2-42b5-9aee-cfdba3875810"). InnerVolumeSpecName "kube-api-access-9c7kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:32:49.907449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.906163    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume\") pod \"ca1817c8-1355-456e-a943-5f5d045df3c6\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") "
Nov 01 06:32:49.907449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.906488    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28\" (UniqueName: \"kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28\") pod \"ca1817c8-1355-456e-a943-5f5d045df3c6\" (UID: \"ca1817c8-1355-456e-a943-5f5d045df3c6\") "
Nov 01 06:32:49.907449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.906963    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7kk\" (UniqueName: \"kubernetes.io/projected/a5d94d75-dde2-42b5-9aee-cfdba3875810-kube-api-access-9c7kk\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:49.907449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.906983    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d94d75-dde2-42b5-9aee-cfdba3875810-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:49.917537 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.908601    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume" pod "ca1817c8-1355-456e-a943-5f5d045df3c6" (UID: "ca1817c8-1355-456e-a943-5f5d045df3c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:32:49.929529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:49.921045    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28" pod "ca1817c8-1355-456e-a943-5f5d045df3c6" (UID: "ca1817c8-1355-456e-a943-5f5d045df3c6"). InnerVolumeSpecName "kube-api-access-jpc28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:32:50.008360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.008326    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpc28\" (UniqueName: \"kubernetes.io/projected/ca1817c8-1355-456e-a943-5f5d045df3c6-kube-api-access-jpc28\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:50.008360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.008357    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1817c8-1355-456e-a943-5f5d045df3c6-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:50.013861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.013833    4468 generic.go:356] "Generic (PLEG): container finished" podID="a5d94d75-dde2-42b5-9aee-cfdba3875810" containerID="76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578" exitCode=0
Nov 01 06:32:50.014015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.013887    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"
Nov 01 06:32:50.014015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.013924    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9" event={"ID":"a5d94d75-dde2-42b5-9aee-cfdba3875810","Type":"ContainerDied","Data":"76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578"}
Nov 01 06:32:50.014015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.013994    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9" event={"ID":"a5d94d75-dde2-42b5-9aee-cfdba3875810","Type":"ContainerDied","Data":"a6cf64b59f29f15d0fae969104450906b7e190cb963fafd6495d5fac6f090bf0"}
Nov 01 06:32:50.014150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.014026    4468 scope.go:122] "RemoveContainer" containerID="76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578"
Nov 01 06:32:50.015541 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.015516    4468 generic.go:356] "Generic (PLEG): container finished" podID="ca1817c8-1355-456e-a943-5f5d045df3c6" containerID="b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330" exitCode=0
Nov 01 06:32:50.015640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.015566    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl" event={"ID":"ca1817c8-1355-456e-a943-5f5d045df3c6","Type":"ContainerDied","Data":"b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330"}
Nov 01 06:32:50.015640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.015582    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"
Nov 01 06:32:50.015640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.015609    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl" event={"ID":"ca1817c8-1355-456e-a943-5f5d045df3c6","Type":"ContainerDied","Data":"1a49618513a78677ba5bce4d18e524403a93c1f63f4e1f2cd4e96b9090d24f38"}
Nov 01 06:32:50.026415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.026392    4468 scope.go:122] "RemoveContainer" containerID="76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578"
Nov 01 06:32:50.026721 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:32:50.026701    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578\": container with ID starting with 76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578 not found: ID does not exist" containerID="76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578"
Nov 01 06:32:50.026782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.026731    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578"} err="failed to get container status \"76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578\": rpc error: code = NotFound desc = could not find container \"76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578\": container with ID starting with 76a70786d0b8bc25918ae94fb9a3b608d79d1454cf3b8fcd29f88ddb98d69578 not found: ID does not exist"
Nov 01 06:32:50.026782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.026748    4468 scope.go:122] "RemoveContainer" containerID="b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330"
Nov 01 06:32:50.036534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.036514    4468 scope.go:122] "RemoveContainer" containerID="b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330"
Nov 01 06:32:50.036819 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:32:50.036800    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330\": container with ID starting with b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330 not found: ID does not exist" containerID="b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330"
Nov 01 06:32:50.036874 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.036829    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330"} err="failed to get container status \"b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330\": rpc error: code = NotFound desc = could not find container \"b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330\": container with ID starting with b7359383178a90e18a89dfb5b26cd8b8fb6b7e135611bae9ffe0daf7180c7330 not found: ID does not exist"
Nov 01 06:32:50.178502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.178373    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"]
Nov 01 06:32:50.184039 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.183980    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-59cb6c8f87-gxkt9"]
Nov 01 06:32:50.282090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.282062    4468 scope.go:122] "RemoveContainer" containerID="6626babce39d9714088fe3f998c47ef1c8e135bccf38181055c6f9d2edb95ba2"
Nov 01 06:32:50.295904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.295885    4468 scope.go:122] "RemoveContainer" containerID="1e2c6e8cc9c9f28779bc3bead20738f67ecf2d8ada0bac3d7c24108626339a90"
Nov 01 06:32:50.295998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.295987    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"]
Nov 01 06:32:50.297527 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.297503    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-66d8c6b55b-zb2pl"]
Nov 01 06:32:50.305631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.305616    4468 scope.go:122] "RemoveContainer" containerID="db637eba3130ac946861d96baae7731b62e8beb18e62604f6430d78b3da03e90"
Nov 01 06:32:50.316021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.315999    4468 scope.go:122] "RemoveContainer" containerID="7cea31c20ab92255bd1649ec8251fb2bbc74d69b54270238457b9e4ce1307579"
Nov 01 06:32:50.326352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.326334    4468 scope.go:122] "RemoveContainer" containerID="675bdfc4c4e6687feec0e6ad056b9c4e1827a8600aa2749020f9235c54b989a7"
Nov 01 06:32:50.335614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.335592    4468 scope.go:122] "RemoveContainer" containerID="2a2e21f30a4d26fc3e5e81c3769f24377c2f8762406b6b69d8297ccfe373bd49"
Nov 01 06:32:50.590637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.590609    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d94d75-dde2-42b5-9aee-cfdba3875810" path="/var/lib/kubelet/pods/a5d94d75-dde2-42b5-9aee-cfdba3875810/volumes"
Nov 01 06:32:50.591021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:50.591008    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca1817c8-1355-456e-a943-5f5d045df3c6" path="/var/lib/kubelet/pods/ca1817c8-1355-456e-a943-5f5d045df3c6/volumes"
Nov 01 06:32:56.047076 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:56.047039    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"]
Nov 01 06:32:56.094932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:56.094897    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-818428b6-x2zng"]
Nov 01 06:32:56.486411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:56.486330    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"]
Nov 01 06:32:56.486620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:56.486596    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy" containerID="cri-o://e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62" gracePeriod=30
Nov 01 06:32:56.651680 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:32:56.644709    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b388dc36-3146-41bc-9ba9-59023025de3c" path="/var/lib/kubelet/pods/b388dc36-3146-41bc-9ba9-59023025de3c/volumes"
Nov 01 06:33:01.742051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.742024    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:33:01.790789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790707    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.790789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790774    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790828    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790867    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790903    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790921    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.790939    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791075    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:01.791277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791122    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791170    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98\") pod \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\" (UID: \"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c\") "
Nov 01 06:33:01.791737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791315    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:01.791737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791430    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:01.791737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791430    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:01.792147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.791734    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:01.792147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.792006    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.792147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.792026    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.792147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.792041    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.792147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.792058    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.792147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.792073    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.793354 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.793326    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:01.793621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.793403    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:33:01.793621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.793437    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:01.793621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.793579    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98" pod "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" (UID: "f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c"). InnerVolumeSpecName "kube-api-access-qpc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:01.893251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.893219    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.893251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.893250    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpc98\" (UniqueName: \"kubernetes.io/projected/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-kube-api-access-qpc98\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.893251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.893260    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:01.893733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:01.893271    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:02.097844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.097804    4468 generic.go:356] "Generic (PLEG): container finished" podID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerID="e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62" exitCode=0
Nov 01 06:33:02.098029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.097858    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"
Nov 01 06:33:02.098029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.097894    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" event={"ID":"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c","Type":"ContainerDied","Data":"e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62"}
Nov 01 06:33:02.098029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.097963    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw" event={"ID":"f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c","Type":"ContainerDied","Data":"e22c5092ce56b15a61c1897afe698043373ff8f04ff76cbdba3e343f775266f6"}
Nov 01 06:33:02.098029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.097992    4468 scope.go:122] "RemoveContainer" containerID="e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62"
Nov 01 06:33:02.108773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.108750    4468 scope.go:122] "RemoveContainer" containerID="498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"
Nov 01 06:33:02.118416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.118396    4468 scope.go:122] "RemoveContainer" containerID="e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62"
Nov 01 06:33:02.118804 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:33:02.118781    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62\": container with ID starting with e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62 not found: ID does not exist" containerID="e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62"
Nov 01 06:33:02.118864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.118825    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62"} err="failed to get container status \"e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62\": rpc error: code = NotFound desc = could not find container \"e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62\": container with ID starting with e87bc7d29348211f917965894a14bf50c8360b981f0ff6670fce56143197ae62 not found: ID does not exist"
Nov 01 06:33:02.118864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.118849    4468 scope.go:122] "RemoveContainer" containerID="498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"
Nov 01 06:33:02.119117 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:33:02.119096    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a\": container with ID starting with 498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a not found: ID does not exist" containerID="498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"
Nov 01 06:33:02.119187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.119123    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a"} err="failed to get container status \"498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a\": rpc error: code = NotFound desc = could not find container \"498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a\": container with ID starting with 498e066fc46d609cc1918db088e9d331791c66fd2c40041cd70a1c9324a5457a not found: ID does not exist"
Nov 01 06:33:02.276662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.276614    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"]
Nov 01 06:33:02.280298 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.280274    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd9d24-openshift-default-6774c4b897-crvgw"]
Nov 01 06:33:02.590161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:02.590126    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" path="/var/lib/kubelet/pods/f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c/volumes"
Nov 01 06:33:02.948131 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:33:02.948048    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:33:06.150459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:06.150412    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:33:13.044548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:13.044499    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"]
Nov 01 06:33:13.093746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:13.093711    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45361c63-ljt6b"]
Nov 01 06:33:14.590875 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:14.590841    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce6df7b-e3ac-484e-8b71-8f72a11318e5" path="/var/lib/kubelet/pods/3ce6df7b-e3ac-484e-8b71-8f72a11318e5/volumes"
Nov 01 06:33:15.480116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.480082    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"]
Nov 01 06:33:15.480369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.480320    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x" podUID="0a783741-283e-4888-9ec0-2d5aa66d8607" containerName="payload-processing" containerID="cri-o://70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa" gracePeriod=30
Nov 01 06:33:15.531848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.531809    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"]
Nov 01 06:33:15.532049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.532013    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq" podUID="ba6df96d-ad9c-4962-8c36-8ed0cdfab191" containerName="payload-pre-processing" containerID="cri-o://8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9" gracePeriod=30
Nov 01 06:33:15.789548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.789515    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:33:15.825361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.825337    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:33:15.874386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874347    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume\") pod \"0a783741-283e-4888-9ec0-2d5aa66d8607\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") "
Nov 01 06:33:15.874588 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874398    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg\") pod \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") "
Nov 01 06:33:15.874588 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874518    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume\") pod \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\" (UID: \"ba6df96d-ad9c-4962-8c36-8ed0cdfab191\") "
Nov 01 06:33:15.874588 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874555    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz\" (UniqueName: \"kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz\") pod \"0a783741-283e-4888-9ec0-2d5aa66d8607\" (UID: \"0a783741-283e-4888-9ec0-2d5aa66d8607\") "
Nov 01 06:33:15.874878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874851    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume" pod "0a783741-283e-4888-9ec0-2d5aa66d8607" (UID: "0a783741-283e-4888-9ec0-2d5aa66d8607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:15.874939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874881    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume" pod "ba6df96d-ad9c-4962-8c36-8ed0cdfab191" (UID: "ba6df96d-ad9c-4962-8c36-8ed0cdfab191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:15.874992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874979    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.875025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.874995    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a783741-283e-4888-9ec0-2d5aa66d8607-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.876737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.876718    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz" pod "0a783741-283e-4888-9ec0-2d5aa66d8607" (UID: "0a783741-283e-4888-9ec0-2d5aa66d8607"). InnerVolumeSpecName "kube-api-access-n5xzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:15.876818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.876798    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg" pod "ba6df96d-ad9c-4962-8c36-8ed0cdfab191" (UID: "ba6df96d-ad9c-4962-8c36-8ed0cdfab191"). InnerVolumeSpecName "kube-api-access-66dzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:15.976046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.976012    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/ba6df96d-ad9c-4962-8c36-8ed0cdfab191-kube-api-access-66dzg\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.976046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:15.976039    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xzz\" (UniqueName: \"kubernetes.io/projected/0a783741-283e-4888-9ec0-2d5aa66d8607-kube-api-access-n5xzz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:16.194975 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.194944    4468 generic.go:356] "Generic (PLEG): container finished" podID="0a783741-283e-4888-9ec0-2d5aa66d8607" containerID="70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa" exitCode=0
Nov 01 06:33:16.195154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.194998    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"
Nov 01 06:33:16.195154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.195025    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x" event={"ID":"0a783741-283e-4888-9ec0-2d5aa66d8607","Type":"ContainerDied","Data":"70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa"}
Nov 01 06:33:16.195154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.195079    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x" event={"ID":"0a783741-283e-4888-9ec0-2d5aa66d8607","Type":"ContainerDied","Data":"133433fe57ac9aa518e52d72662b346da8d6abed51e76be287d81b442acbc58a"}
Nov 01 06:33:16.195154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.195106    4468 scope.go:122] "RemoveContainer" containerID="70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa"
Nov 01 06:33:16.196612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.196589    4468 generic.go:356] "Generic (PLEG): container finished" podID="ba6df96d-ad9c-4962-8c36-8ed0cdfab191" containerID="8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9" exitCode=0
Nov 01 06:33:16.196716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.196636    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"
Nov 01 06:33:16.196716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.196680    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq" event={"ID":"ba6df96d-ad9c-4962-8c36-8ed0cdfab191","Type":"ContainerDied","Data":"8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9"}
Nov 01 06:33:16.196829 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.196723    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq" event={"ID":"ba6df96d-ad9c-4962-8c36-8ed0cdfab191","Type":"ContainerDied","Data":"8c0ad91cf9b2982e6587a7f3e112914c5a1fe78ae5d0465c014bc9e6b6ca0cf4"}
Nov 01 06:33:16.207190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.207162    4468 scope.go:122] "RemoveContainer" containerID="70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa"
Nov 01 06:33:16.207524 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:33:16.207504    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa\": container with ID starting with 70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa not found: ID does not exist" containerID="70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa"
Nov 01 06:33:16.207579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.207536    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa"} err="failed to get container status \"70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa\": rpc error: code = NotFound desc = could not find container \"70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa\": container with ID starting with 70ad463a0b76b1a1143e5aaf090f672b653854444e1f485ed6db30e1502c76aa not found: ID does not exist"
Nov 01 06:33:16.207579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.207554    4468 scope.go:122] "RemoveContainer" containerID="8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9"
Nov 01 06:33:16.217915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.217895    4468 scope.go:122] "RemoveContainer" containerID="8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9"
Nov 01 06:33:16.218193 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:33:16.218173    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9\": container with ID starting with 8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9 not found: ID does not exist" containerID="8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9"
Nov 01 06:33:16.218281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.218202    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9"} err="failed to get container status \"8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9\": rpc error: code = NotFound desc = could not find container \"8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9\": container with ID starting with 8b0722a26cf242f367c06d5380f4451a88d78a8e543b8566c964c47e9bf681d9 not found: ID does not exist"
Nov 01 06:33:16.362767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.362733    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"]
Nov 01 06:33:16.367916 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.367890    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-558bc9799b-6844x"]
Nov 01 06:33:16.469714 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.469631    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"]
Nov 01 06:33:16.472819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.472798    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-645fd85787-ndlnq"]
Nov 01 06:33:16.590671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.590641    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a783741-283e-4888-9ec0-2d5aa66d8607" path="/var/lib/kubelet/pods/0a783741-283e-4888-9ec0-2d5aa66d8607/volumes"
Nov 01 06:33:16.591062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:16.591049    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6df96d-ad9c-4962-8c36-8ed0cdfab191" path="/var/lib/kubelet/pods/ba6df96d-ad9c-4962-8c36-8ed0cdfab191/volumes"
Nov 01 06:33:32.948295 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:33:32.948259    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:33:38.711091 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:38.711059    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:33:50.396530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:50.396496    4468 scope.go:122] "RemoveContainer" containerID="cad0c361c9a745500d39d1b9df4768dab00e5fe14cb1d364caba0614913cd7b8"
Nov 01 06:33:54.045744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:54.045712    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"]
Nov 01 06:33:54.095400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:54.095365    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d010850d-cjbgk"]
Nov 01 06:33:54.590682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:33:54.590650    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f499dd-81b4-4cef-98c5-81e15cdf8c29" path="/var/lib/kubelet/pods/27f499dd-81b4-4cef-98c5-81e15cdf8c29/volumes"
Nov 01 06:34:00.044819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:00.044786    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"]
Nov 01 06:34:00.101163 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:00.095904    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ef364b2-tg562"]
Nov 01 06:34:00.590344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:00.590312    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0acf3f5-6233-422f-b186-96d1a3c53be7" path="/var/lib/kubelet/pods/b0acf3f5-6233-422f-b186-96d1a3c53be7/volumes"
Nov 01 06:34:02.947761 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:34:02.947715    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:34:05.393523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.393489    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"]
Nov 01 06:34:05.394022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394009    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a5d94d75-dde2-42b5-9aee-cfdba3875810" containerName="payload-pre-processing"
Nov 01 06:34:05.394068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394024    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d94d75-dde2-42b5-9aee-cfdba3875810" containerName="payload-pre-processing"
Nov 01 06:34:05.394068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394039    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy"
Nov 01 06:34:05.394068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394045    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy"
Nov 01 06:34:05.394068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394062    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" containerName="revoke-keys"
Nov 01 06:34:05.394068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394067    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" containerName="revoke-keys"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394076    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0a783741-283e-4888-9ec0-2d5aa66d8607" containerName="payload-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394080    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a783741-283e-4888-9ec0-2d5aa66d8607" containerName="payload-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394088    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ba6df96d-ad9c-4962-8c36-8ed0cdfab191" containerName="payload-pre-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394093    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6df96d-ad9c-4962-8c36-8ed0cdfab191" containerName="payload-pre-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394102    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ca1817c8-1355-456e-a943-5f5d045df3c6" containerName="payload-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394106    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1817c8-1355-456e-a943-5f5d045df3c6" containerName="payload-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394197    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1817c8-1355-456e-a943-5f5d045df3c6" containerName="payload-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394207    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394216    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d94d75-dde2-42b5-9aee-cfdba3875810" containerName="payload-pre-processing"
Nov 01 06:34:05.394218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394222    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy"
Nov 01 06:34:05.394533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394232    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" containerName="revoke-keys"
Nov 01 06:34:05.394533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394237    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a783741-283e-4888-9ec0-2d5aa66d8607" containerName="payload-processing"
Nov 01 06:34:05.394533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.394242    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6df96d-ad9c-4962-8c36-8ed0cdfab191" containerName="payload-pre-processing"
Nov 01 06:34:05.398942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.398926    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.487790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.487759    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.487936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.487814    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.487936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.487842    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.487936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.487899    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrl5\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.488042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.487952    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.488076 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.488046    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.488076 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.488069    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.488141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.488085    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.488141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.488112    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.504875 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.504841    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"]
Nov 01 06:34:05.588986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.588955    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.588986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.588991    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589007    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589035    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589118    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589180    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589208    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589231    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrl5\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589295    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589404    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589489    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589566 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589519    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.589926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.589907    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.590213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.590191    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.591948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.591920    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.592111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.592092    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.645399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.645317    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.645399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.645363    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrl5\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5\") pod \"maas-default-gateway-openshift-default-7d9897b54c-p4lr6\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:05.712254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:05.712222    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:06.140716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.140690    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"]
Nov 01 06:34:06.141717 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:34:06.141686    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd4b0b1_ce47_419d_a68f_1dc75422da8f.slice/crio-1c30b3ff7bdceb6810994ddf0633d42c91d90e3fa2a2c82dfe77da95fb68c336 WatchSource:0}: Error finding container 1c30b3ff7bdceb6810994ddf0633d42c91d90e3fa2a2c82dfe77da95fb68c336: Status 404 returned error can't find the container with id 1c30b3ff7bdceb6810994ddf0633d42c91d90e3fa2a2c82dfe77da95fb68c336
Nov 01 06:34:06.144199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.144162    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:06.144276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.144257    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:06.144321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.144286    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:06.528586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.528506    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" event={"ID":"ccd4b0b1-ce47-419d-a68f-1dc75422da8f","Type":"ContainerStarted","Data":"a1051f1c1bf399d18add31ba8fb58be7b579d53c00c9998c2b9ad98a2ddcee92"}
Nov 01 06:34:06.528586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.528566    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" event={"ID":"ccd4b0b1-ce47-419d-a68f-1dc75422da8f","Type":"ContainerStarted","Data":"1c30b3ff7bdceb6810994ddf0633d42c91d90e3fa2a2c82dfe77da95fb68c336"}
Nov 01 06:34:06.645968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.645905    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podStartSLOduration=1.645883908 podStartE2EDuration="1.645883908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:06.642369705 +0000 UTC m=+1458.210176561" watchObservedRunningTime="2026-11-01 06:34:06.645883908 +0000 UTC m=+1458.213690738"
Nov 01 06:34:06.720490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:06.715553    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:07.725496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:07.720865    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:07.725496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:07.720936    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:08.726881 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:08.726842    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:08.728586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:08.728550    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:09.723492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:09.713856    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:09.723492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:09.713931    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:10.046894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:10.046862    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"]
Nov 01 06:34:10.095443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:10.095415    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e21ccfc-qhlhc"]
Nov 01 06:34:10.590204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:10.590175    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa42faa-aa32-4d8b-a46f-cdd186278093" path="/var/lib/kubelet/pods/2fa42faa-aa32-4d8b-a46f-cdd186278093/volumes"
Nov 01 06:34:10.716862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:10.716825    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:10.717123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:10.717091    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:11.722492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:11.717823    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:11.722492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:11.717881    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:11.951000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:11.950968    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"]
Nov 01 06:34:11.951521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:11.951507    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy"
Nov 01 06:34:11.951573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:11.951522    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fa8b9a-2e39-4c44-9d6e-e72073a42b6c" containerName="istio-proxy"
Nov 01 06:34:11.955130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:11.955113    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.030501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.030382    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"]
Nov 01 06:34:12.058531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.058496    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:34:12.058531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.058496    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:34:12.058531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.058498    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:34:12.058531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.058535    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:34:12.092402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.092375    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5rv\" (UniqueName: \"kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.092540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.092408    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.092540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.092445    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.193968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.193927    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.194165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.194109    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5rv\" (UniqueName: \"kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.194165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.194136    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.194768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.194745    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.196407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.196386    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.250393 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.250365    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5rv\" (UniqueName: \"kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv\") pod \"maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.268962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.268933    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:12.644883 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.644857    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"]
Nov 01 06:34:12.646205 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:34:12.646180    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609c50ed_e7ce_406c_8c0a_96a60ade0ebd.slice/crio-a659898db335ecf3b28701d059813e3cdd0ff2e6637a57ca978cc08d2c9be03e WatchSource:0}: Error finding container a659898db335ecf3b28701d059813e3cdd0ff2e6637a57ca978cc08d2c9be03e: Status 404 returned error can't find the container with id a659898db335ecf3b28701d059813e3cdd0ff2e6637a57ca978cc08d2c9be03e
Nov 01 06:34:12.713550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.713513    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:12.713665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:12.713578    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:13.488012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.487979    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"]
Nov 01 06:34:13.492726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.492707    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.547838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.547805    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"]
Nov 01 06:34:13.580698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.580658    4468 generic.go:356] "Generic (PLEG): container finished" podID="609c50ed-e7ce-406c-8c0a-96a60ade0ebd" containerID="cb50ed9a13a85f37f60ded3ebc392262a6da53bb96a1ea3a671dce0dba5fc66c" exitCode=0
Nov 01 06:34:13.580859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.580735    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7" event={"ID":"609c50ed-e7ce-406c-8c0a-96a60ade0ebd","Type":"ContainerDied","Data":"cb50ed9a13a85f37f60ded3ebc392262a6da53bb96a1ea3a671dce0dba5fc66c"}
Nov 01 06:34:13.580859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.580812    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7" event={"ID":"609c50ed-e7ce-406c-8c0a-96a60ade0ebd","Type":"ContainerStarted","Data":"a659898db335ecf3b28701d059813e3cdd0ff2e6637a57ca978cc08d2c9be03e"}
Nov 01 06:34:13.612429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.612394    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvd8n\" (UniqueName: \"kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.612586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.612554    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.612641 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.612611    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.713010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.712974    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 06:34:13.713217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.713031    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 06:34:13.713701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.713674    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvd8n\" (UniqueName: \"kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.713823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.713740    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.713823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.713806    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.714693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.714668    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.716962 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.716927    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.772586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.772505    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvd8n\" (UniqueName: \"kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n\") pod \"maas-api-revoke-sub-bd3123b6-pxp59\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:13.806477 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:13.806433    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:14.193860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.193742    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"]
Nov 01 06:34:14.593358 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.593321    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59" event={"ID":"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6","Type":"ContainerStarted","Data":"347f7394ee654f7e4695c9b74436e456881e20c967161c7e3ff8c56bd7cbdfe2"}
Nov 01 06:34:14.593752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.593386    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59" event={"ID":"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6","Type":"ContainerStarted","Data":"aa4c423a0dd35394fd8aecafa58f4c4aa39ffae22f0bcbff739c969245ac9855"}
Nov 01 06:34:14.705740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.705674    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59" podStartSLOduration=1.705653683 podStartE2EDuration="1.705653683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:14.701139522 +0000 UTC m=+1466.268946353" watchObservedRunningTime="2026-11-01 06:34:14.705653683 +0000 UTC m=+1466.273460514"
Nov 01 06:34:14.717148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.717120    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:14.717455 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.717432    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:14.718149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.718131    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 06:34:14.751436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.751418    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:14.830865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.830811    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca\") pod \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") "
Nov 01 06:34:14.831137 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.830973    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv\" (UniqueName: \"kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv\") pod \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") "
Nov 01 06:34:14.831137 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.831063    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca\") pod \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\" (UID: \"609c50ed-e7ce-406c-8c0a-96a60ade0ebd\") "
Nov 01 06:34:14.833125 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.833095    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca" pod "609c50ed-e7ce-406c-8c0a-96a60ade0ebd" (UID: "609c50ed-e7ce-406c-8c0a-96a60ade0ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:34:14.833943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.833915    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv" pod "609c50ed-e7ce-406c-8c0a-96a60ade0ebd" (UID: "609c50ed-e7ce-406c-8c0a-96a60ade0ebd"). InnerVolumeSpecName "kube-api-access-sl5rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:34:14.834047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.834031    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca" pod "609c50ed-e7ce-406c-8c0a-96a60ade0ebd" (UID: "609c50ed-e7ce-406c-8c0a-96a60ade0ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:34:14.933244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.933211    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:14.933244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.933242    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5rv\" (UniqueName: \"kubernetes.io/projected/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-kube-api-access-sl5rv\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:14.933421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:14.933253    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609c50ed-e7ce-406c-8c0a-96a60ade0ebd-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:15.596957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:15.596933    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"
Nov 01 06:34:15.596957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:15.596950    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7" event={"ID":"609c50ed-e7ce-406c-8c0a-96a60ade0ebd","Type":"ContainerDied","Data":"a659898db335ecf3b28701d059813e3cdd0ff2e6637a57ca978cc08d2c9be03e"}
Nov 01 06:34:15.597437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:15.596986    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a659898db335ecf3b28701d059813e3cdd0ff2e6637a57ca978cc08d2c9be03e"
Nov 01 06:34:15.598833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:15.598790    4468 generic.go:356] "Generic (PLEG): container finished" podID="d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" containerID="347f7394ee654f7e4695c9b74436e456881e20c967161c7e3ff8c56bd7cbdfe2" exitCode=0
Nov 01 06:34:15.598959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:15.598903    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59" event={"ID":"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6","Type":"ContainerDied","Data":"347f7394ee654f7e4695c9b74436e456881e20c967161c7e3ff8c56bd7cbdfe2"}
Nov 01 06:34:16.746719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.746693    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:16.856967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.856895    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca\") pod \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") "
Nov 01 06:34:16.856967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.856954    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca\") pod \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") "
Nov 01 06:34:16.857136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.857101    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n\" (UniqueName: \"kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n\") pod \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\" (UID: \"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6\") "
Nov 01 06:34:16.857233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.857213    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca" pod "d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" (UID: "d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:34:16.857398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.857379    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:16.859607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.859579    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n" pod "d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" (UID: "d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6"). InnerVolumeSpecName "kube-api-access-lvd8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:34:16.859781 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.859757    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca" pod "d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" (UID: "d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:34:16.958824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.958794    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvd8n\" (UniqueName: \"kubernetes.io/projected/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-kube-api-access-lvd8n\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:16.958824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:16.958819    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:17.612640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:17.612604    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"
Nov 01 06:34:17.612807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:17.612637    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59" event={"ID":"d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6","Type":"ContainerDied","Data":"aa4c423a0dd35394fd8aecafa58f4c4aa39ffae22f0bcbff739c969245ac9855"}
Nov 01 06:34:17.612807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:17.612684    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4c423a0dd35394fd8aecafa58f4c4aa39ffae22f0bcbff739c969245ac9855"
Nov 01 06:34:17.850173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:17.850133    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:34:20.082153 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:20.082124    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"]
Nov 01 06:34:20.131452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:20.131425    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec319567-vj22n"]
Nov 01 06:34:20.590086 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:20.590055    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89992b30-590b-46f0-9b47-602062b4fc10" path="/var/lib/kubelet/pods/89992b30-590b-46f0-9b47-602062b4fc10/volumes"
Nov 01 06:34:32.947035 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:34:32.946999    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:34:38.047634 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:38.047591    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"]
Nov 01 06:34:38.094321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:38.094283    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc6dc2c-sjmvj"]
Nov 01 06:34:38.592000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:38.591968    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96280be4-3a6c-4f5c-bdae-87316a0e5e92" path="/var/lib/kubelet/pods/96280be4-3a6c-4f5c-bdae-87316a0e5e92/volumes"
Nov 01 06:34:45.760110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760077    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"]
Nov 01 06:34:45.760631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760617    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" containerName="revoke-keys"
Nov 01 06:34:45.760678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760632    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" containerName="revoke-keys"
Nov 01 06:34:45.760678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760657    4468 cpu_manager.go:414] "containerMap: removing container" podUID="609c50ed-e7ce-406c-8c0a-96a60ade0ebd" containerName="revoke-keys"
Nov 01 06:34:45.760678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760662    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609c50ed-e7ce-406c-8c0a-96a60ade0ebd" containerName="revoke-keys"
Nov 01 06:34:45.760779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760752    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" containerName="revoke-keys"
Nov 01 06:34:45.760779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.760765    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="609c50ed-e7ce-406c-8c0a-96a60ade0ebd" containerName="revoke-keys"
Nov 01 06:34:45.764368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.764353    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:45.824105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.824066    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"]
Nov 01 06:34:45.869838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.869757    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:34:45.869838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.869792    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:34:45.869838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.869798    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:34:45.869838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.869816    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:34:45.906042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.906014    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:45.906216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.906055    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:45.906216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:45.906195    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhmg\" (UniqueName: \"kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.007101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.007074    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.007276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.007121    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.007276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.007195    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhmg\" (UniqueName: \"kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.007719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.007700    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.009553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.009531    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.063513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.063459    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhmg\" (UniqueName: \"kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg\") pod \"maas-api-revoke-sub-0e84b520-h6jmz\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.086318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.086295    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:46.466392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.466311    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"]
Nov 01 06:34:46.471759 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:34:46.471725    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18caf57d_919f_48b9_be20_de7e9e5139d4.slice/crio-31d5b5d5a7fac0d5921b75439d59094034bd08dbdc3eab3c27e5a357fe287b50 WatchSource:0}: Error finding container 31d5b5d5a7fac0d5921b75439d59094034bd08dbdc3eab3c27e5a357fe287b50: Status 404 returned error can't find the container with id 31d5b5d5a7fac0d5921b75439d59094034bd08dbdc3eab3c27e5a357fe287b50
Nov 01 06:34:46.812538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:46.812506    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz" event={"ID":"18caf57d-919f-48b9-be20-de7e9e5139d4","Type":"ContainerStarted","Data":"31d5b5d5a7fac0d5921b75439d59094034bd08dbdc3eab3c27e5a357fe287b50"}
Nov 01 06:34:47.825513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:47.825460    4468 generic.go:356] "Generic (PLEG): container finished" podID="18caf57d-919f-48b9-be20-de7e9e5139d4" containerID="94ff860ea220b6d7c004284cffb609257d567c38f6a833d8206f0c0da9204796" exitCode=0
Nov 01 06:34:47.826056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:47.825566    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz" event={"ID":"18caf57d-919f-48b9-be20-de7e9e5139d4","Type":"ContainerDied","Data":"94ff860ea220b6d7c004284cffb609257d567c38f6a833d8206f0c0da9204796"}
Nov 01 06:34:48.978550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:48.978517    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:49.065633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.065602    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca\") pod \"18caf57d-919f-48b9-be20-de7e9e5139d4\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") "
Nov 01 06:34:49.065823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.065793    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca\") pod \"18caf57d-919f-48b9-be20-de7e9e5139d4\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") "
Nov 01 06:34:49.065823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.065811    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg\" (UniqueName: \"kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg\") pod \"18caf57d-919f-48b9-be20-de7e9e5139d4\" (UID: \"18caf57d-919f-48b9-be20-de7e9e5139d4\") "
Nov 01 06:34:49.066164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.066140    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca" pod "18caf57d-919f-48b9-be20-de7e9e5139d4" (UID: "18caf57d-919f-48b9-be20-de7e9e5139d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:34:49.067942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.067913    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca" pod "18caf57d-919f-48b9-be20-de7e9e5139d4" (UID: "18caf57d-919f-48b9-be20-de7e9e5139d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:34:49.068072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.068045    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg" pod "18caf57d-919f-48b9-be20-de7e9e5139d4" (UID: "18caf57d-919f-48b9-be20-de7e9e5139d4"). InnerVolumeSpecName "kube-api-access-fxhmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:34:49.084168 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.084101    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"]
Nov 01 06:34:49.087227 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.087205    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23734bcc-7qmt2"]
Nov 01 06:34:49.167450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.167422    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:49.167450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.167445    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18caf57d-919f-48b9-be20-de7e9e5139d4-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:49.167450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.167455    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxhmg\" (UniqueName: \"kubernetes.io/projected/18caf57d-919f-48b9-be20-de7e9e5139d4-kube-api-access-fxhmg\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:34:49.840515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.840442    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz" event={"ID":"18caf57d-919f-48b9-be20-de7e9e5139d4","Type":"ContainerDied","Data":"31d5b5d5a7fac0d5921b75439d59094034bd08dbdc3eab3c27e5a357fe287b50"}
Nov 01 06:34:49.840515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.840507    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"
Nov 01 06:34:49.840723 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:49.840524    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d5b5d5a7fac0d5921b75439d59094034bd08dbdc3eab3c27e5a357fe287b50"
Nov 01 06:34:50.435389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.435358    4468 scope.go:122] "RemoveContainer" containerID="2048b098b966ba9a5d65c0946cd438bb48b4ba3201abecaef77efe30adc31ab0"
Nov 01 06:34:50.447304 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.447286    4468 scope.go:122] "RemoveContainer" containerID="6d2fbb864e8c57e3cf0fdcbf8d2e3843c0adc151421a8afc2c4b53a7977ffeb3"
Nov 01 06:34:50.590331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.590298    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99756e7-4423-40ec-91d3-66be07388017" path="/var/lib/kubelet/pods/c99756e7-4423-40ec-91d3-66be07388017/volumes"
Nov 01 06:34:50.747712 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.747636    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"]
Nov 01 06:34:50.748186 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.748173    4468 cpu_manager.go:414] "containerMap: removing container" podUID="18caf57d-919f-48b9-be20-de7e9e5139d4" containerName="revoke-keys"
Nov 01 06:34:50.748231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.748187    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18caf57d-919f-48b9-be20-de7e9e5139d4" containerName="revoke-keys"
Nov 01 06:34:50.748276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.748268    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="18caf57d-919f-48b9-be20-de7e9e5139d4" containerName="revoke-keys"
Nov 01 06:34:50.752899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.752881    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.797218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.797183    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"]
Nov 01 06:34:50.801151 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.801132    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.871894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.871865    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"]
Nov 01 06:34:50.889066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889036    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnn8\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889263 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889178    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889263 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889242    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889381 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889268    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889381 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889370    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889409    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889435    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889608    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.889667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.889646    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.936219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.936187    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"]
Nov 01 06:34:50.990665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990628    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.990665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990669    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.990932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990713    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.990932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990784    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.990932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990842    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.990932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990888    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.990932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.990925    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991018    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.991193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991083    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991132    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991272    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.991344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991313    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991394    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991421    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991428    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.991663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991444    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991457    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.991663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991564    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.991663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991615    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnn8\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991629    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991677    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.991873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991775    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.991873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.991806    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6b4\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:50.993767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.993743    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:50.993995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:50.993974    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:51.047856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.047827    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:51.048030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.047993    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnn8\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fxl5k\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:51.066120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.066093    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:51.092788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.092754    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.092938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.092825    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.092938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.092849    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.092938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.092878    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.092938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.092927    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.092953    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6b4\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093043    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093161    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093201    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093203    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093279    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093697    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093775    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.093866 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.093845    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.095154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.095136    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.095751 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.095722    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.149283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.149241    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6b4\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.149671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.149645    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-n2vdj\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.423132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.423043    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:51.498981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.498174    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"]
Nov 01 06:34:51.500235 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:34:51.500206    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b84a1c_36e2_44c7_9074_289cbc3e2698.slice/crio-63ff8de268b938207c483b7c8f38ac5db18a33ccd47020e48a6f2c7e5c95445f WatchSource:0}: Error finding container 63ff8de268b938207c483b7c8f38ac5db18a33ccd47020e48a6f2c7e5c95445f: Status 404 returned error can't find the container with id 63ff8de268b938207c483b7c8f38ac5db18a33ccd47020e48a6f2c7e5c95445f
Nov 01 06:34:51.503493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.503426    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:51.503616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.503543    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:51.503616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.503603    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:51.802207 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:34:51.802177    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b983509_8fc0_4982_8991_7ec621e59169.slice/crio-de141bc1da74cc636e43729b8f7c5b4ae1a896b6b1454c0a6075820001587d45 WatchSource:0}: Error finding container de141bc1da74cc636e43729b8f7c5b4ae1a896b6b1454c0a6075820001587d45: Status 404 returned error can't find the container with id de141bc1da74cc636e43729b8f7c5b4ae1a896b6b1454c0a6075820001587d45
Nov 01 06:34:51.802761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.802730    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"]
Nov 01 06:34:51.804932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.804897    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:51.805044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.804979    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:51.805044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.805023    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:34:51.855714 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.855688    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" event={"ID":"8b983509-8fc0-4982-8991-7ec621e59169","Type":"ContainerStarted","Data":"de141bc1da74cc636e43729b8f7c5b4ae1a896b6b1454c0a6075820001587d45"}
Nov 01 06:34:51.857521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.857497    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" event={"ID":"91b84a1c-36e2-44c7-9074-289cbc3e2698","Type":"ContainerStarted","Data":"cdbb8909375599f6fde13ce79414aa89ebd66a06cbfe50746dfe66ddc9cb8e52"}
Nov 01 06:34:51.857635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.857540    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" event={"ID":"91b84a1c-36e2-44c7-9074-289cbc3e2698","Type":"ContainerStarted","Data":"63ff8de268b938207c483b7c8f38ac5db18a33ccd47020e48a6f2c7e5c95445f"}
Nov 01 06:34:51.972587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:51.972517    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podStartSLOduration=1.972496085 podStartE2EDuration="1.972496085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:51.971830643 +0000 UTC m=+1503.539637469" watchObservedRunningTime="2026-11-01 06:34:51.972496085 +0000 UTC m=+1503.540302906"
Nov 01 06:34:52.066961 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:52.066879    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:34:52.068102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:52.068079    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:52.068214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:52.068124    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:52.866759 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:52.866721    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" event={"ID":"8b983509-8fc0-4982-8991-7ec621e59169","Type":"ContainerStarted","Data":"9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f"}
Nov 01 06:34:52.997586 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:52.997527    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podStartSLOduration=2.997509427 podStartE2EDuration="2.997509427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:52.997144455 +0000 UTC m=+1504.564951287" watchObservedRunningTime="2026-11-01 06:34:52.997509427 +0000 UTC m=+1504.565316257"
Nov 01 06:34:53.067078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:53.067045    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:53.067247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:53.067099    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:53.428141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:53.427560    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:34:53.428141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:53.427706    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:53.428141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:53.427748    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:54.067323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:54.067282    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:54.067780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:54.067339    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:54.424966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:54.424879    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:54.425227 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:54.425199    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:55.066815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:55.066773    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:55.067020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:55.066831    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:55.424549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:55.423841    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:55.424549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:55.423908    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:55.956542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:55.956504    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:34:56.073492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:56.070417    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:56.073492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:56.070520    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:56.424907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:56.424868    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:56.425497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:56.425446    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:57.075492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:57.068977    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:57.075492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:57.069047    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:57.432101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:57.432000    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:57.432707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:57.432657    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:58.071495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:58.069262    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:58.071495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:58.069322    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:58.424331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:58.424243    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:58.424331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:58.424305    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:59.066978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:59.066945    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fxl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 01 06:34:59.067368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:59.067000    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 01 06:34:59.424511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:59.424412    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-n2vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 06:34:59.424511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:59.424493    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 06:34:59.501951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:59.501913    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"]
Nov 01 06:34:59.535154 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:34:59.535108    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-56ae52d5-4pgm7"]
Nov 01 06:35:00.536532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.534973    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:35:00.536532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.536366    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:35:00.537724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.537705    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:35:00.593855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.593811    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609c50ed-e7ce-406c-8c0a-96a60ade0ebd" path="/var/lib/kubelet/pods/609c50ed-e7ce-406c-8c0a-96a60ade0ebd/volumes"
Nov 01 06:35:00.752507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.752152    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:35:00.752715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.752690    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:35:00.754813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:00.754755    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:35:02.948335 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:35:02.948300    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:35:23.806927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:23.806889    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"]
Nov 01 06:35:23.807502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:23.807136    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy" containerID="cri-o://fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1" gracePeriod=30
Nov 01 06:35:29.072950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.072879    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:35:29.136425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136395    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.136626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136460    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.136626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136509    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.136741 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136720    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.136804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136789    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.136859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136837    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.136912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136857    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:35:29.136912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136882    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.137018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136922    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.137134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137014    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token\") pod \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\" (UID: \"b48a655c-a6fe-4e33-b42f-e79b37b7c142\") "
Nov 01 06:35:29.137134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.136998    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:35:29.137318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137282    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:29.137413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137350    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:35:29.137413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137403    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:35:29.137878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137789    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.137878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137818    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.137878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137833    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.137878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137846    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.137878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.137859    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.138956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.138934    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:35:29.139066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.139040    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "kube-api-access-7645x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:29.139166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.139145    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:35:29.139522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.139504    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token" pod "b48a655c-a6fe-4e33-b42f-e79b37b7c142" (UID: "b48a655c-a6fe-4e33-b42f-e79b37b7c142"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:29.158361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.158326    4468 generic.go:356] "Generic (PLEG): container finished" podID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerID="fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1" exitCode=0
Nov 01 06:35:29.158553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.158377    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"
Nov 01 06:35:29.158553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.158422    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" event={"ID":"b48a655c-a6fe-4e33-b42f-e79b37b7c142","Type":"ContainerDied","Data":"fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1"}
Nov 01 06:35:29.158553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.158507    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8" event={"ID":"b48a655c-a6fe-4e33-b42f-e79b37b7c142","Type":"ContainerDied","Data":"bd85c2475aa64a59eaa84ff7c747fd95aa9e02b0369d361d94619fa7f1f52ede"}
Nov 01 06:35:29.158553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.158539    4468 scope.go:122] "RemoveContainer" containerID="fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1"
Nov 01 06:35:29.170986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.170962    4468 scope.go:122] "RemoveContainer" containerID="dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"
Nov 01 06:35:29.180917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.180900    4468 scope.go:122] "RemoveContainer" containerID="fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1"
Nov 01 06:35:29.181232 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:35:29.181214    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1\": container with ID starting with fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1 not found: ID does not exist" containerID="fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1"
Nov 01 06:35:29.181308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.181241    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1"} err="failed to get container status \"fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1\": rpc error: code = NotFound desc = could not find container \"fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1\": container with ID starting with fcac01c86df265d6fc211f109a729d3094b0079ff7c8e389cd3590693df0bbf1 not found: ID does not exist"
Nov 01 06:35:29.181308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.181258    4468 scope.go:122] "RemoveContainer" containerID="dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"
Nov 01 06:35:29.181457 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:35:29.181435    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2\": container with ID starting with dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2 not found: ID does not exist" containerID="dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"
Nov 01 06:35:29.181616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.181497    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2"} err="failed to get container status \"dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2\": rpc error: code = NotFound desc = could not find container \"dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2\": container with ID starting with dcd0ed7e8b1517aa27557d6b400774f2cea4b0168b98bb23575bf92b99e4a3d2 not found: ID does not exist"
Nov 01 06:35:29.239225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.239187    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.239225 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.239221    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7645x\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-kube-api-access-7645x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.239430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.239238    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.239430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.239250    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48a655c-a6fe-4e33-b42f-e79b37b7c142-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:29.401839 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.401757    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"]
Nov 01 06:35:29.404837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:29.404812    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7b2a3-openshift-default-578969456f-96nt8"]
Nov 01 06:35:30.591002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:30.590964    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" path="/var/lib/kubelet/pods/b48a655c-a6fe-4e33-b42f-e79b37b7c142/volumes"
Nov 01 06:35:32.948272 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:35:32.948238    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:35:34.044998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:34.044957    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"]
Nov 01 06:35:34.094386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:34.094344    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f71484-xs9s5"]
Nov 01 06:35:34.590944 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:34.590911    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893c890f-8d80-405f-a667-ffbbe0de5917" path="/var/lib/kubelet/pods/893c890f-8d80-405f-a667-ffbbe0de5917/volumes"
Nov 01 06:35:41.044061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:41.044024    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"]
Nov 01 06:35:41.093449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:41.093418    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-630e6309-g842h"]
Nov 01 06:35:42.593011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:42.592967    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633be351-2af8-4cac-aaf9-42817db31ba2" path="/var/lib/kubelet/pods/633be351-2af8-4cac-aaf9-42817db31ba2/volumes"
Nov 01 06:35:49.046513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:49.046438    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"]
Nov 01 06:35:49.095890 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:49.095854    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6a1470-kdtf8"]
Nov 01 06:35:49.350118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:49.350034    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:35:50.476157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.476126    4468 scope.go:122] "RemoveContainer" containerID="fb4c6872cab1de07b69b72c94c6ddb86d391fb9032441a437673e77cbca5d6a9"
Nov 01 06:35:50.489122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.489088    4468 scope.go:122] "RemoveContainer" containerID="d4f43c69c56a3112d46fdff8604931dfc23e53a0b84c3533b9b91eae3335ee51"
Nov 01 06:35:50.498704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.498684    4468 scope.go:122] "RemoveContainer" containerID="82740c020a8264987df6d43edd2989833faa37c13ff7b37c2b3d23db7826a123"
Nov 01 06:35:50.507223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.507206    4468 scope.go:122] "RemoveContainer" containerID="b40f72362c23a62c9313eca690a81611fb325e1c0ec185026a8e2b00dc7801a8"
Nov 01 06:35:50.515742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.515721    4468 scope.go:122] "RemoveContainer" containerID="3b0d41306208e63d6665570e247002bcf11eae933b6f3728be28d47109bde5fe"
Nov 01 06:35:50.524368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.524351    4468 scope.go:122] "RemoveContainer" containerID="e883e4d41dd24c3192958acf36b05b9a29e4cca960b34ab28eedb95d2ee3ea0a"
Nov 01 06:35:50.589877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:50.589846    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b520194f-14ba-4651-95b3-5ba2b06444e3" path="/var/lib/kubelet/pods/b520194f-14ba-4651-95b3-5ba2b06444e3/volumes"
Nov 01 06:35:59.045726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:59.045694    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"]
Nov 01 06:35:59.094999 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:35:59.094968    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937897ad-w26fh"]
Nov 01 06:36:00.590682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:00.590652    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135756d4-93a2-465e-b896-03c577188d78" path="/var/lib/kubelet/pods/135756d4-93a2-465e-b896-03c577188d78/volumes"
Nov 01 06:36:02.947816 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:36:02.947779    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:36:14.044532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:14.044496    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"]
Nov 01 06:36:14.093970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:14.093933    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75e8fe5a-pbqnn"]
Nov 01 06:36:14.590495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:14.590438    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d367d20-645e-4745-b704-194e688ce8df" path="/var/lib/kubelet/pods/3d367d20-645e-4745-b704-194e688ce8df/volumes"
Nov 01 06:36:20.185746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:20.185702    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:36:24.659448 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.659412    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"]
Nov 01 06:36:24.660821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.660791    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy"
Nov 01 06:36:24.661003 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.660988    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy"
Nov 01 06:36:24.661112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.661101    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy"
Nov 01 06:36:24.661194 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.661183    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy"
Nov 01 06:36:24.661706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.661684    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy"
Nov 01 06:36:24.661861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.661849    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48a655c-a6fe-4e33-b42f-e79b37b7c142" containerName="istio-proxy"
Nov 01 06:36:24.673842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.673813    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.775967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.775922    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"]
Nov 01 06:36:24.792293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792244    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792358    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792402    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792436    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792460    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxwt\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792555    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792648    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792689    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.792736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.792734    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.812336 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.812305    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2e0116-openshift-default-dockercfg-mkvwh\""
Nov 01 06:36:24.894137 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.894092    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.895693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.894153    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.895693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.894184    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.895693 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.894205    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxwt\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.895939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.895733    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.896415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.896387    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.896581 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.894258    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.897035    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897537 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.896686    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897658 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.897568    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897658 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.897590    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.897684    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.897814    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.897922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.897903    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.898603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.898579    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.898833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.898811    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.950925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.950857    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxwt\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:24.953789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:24.953766    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token\") pod \"e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") " pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:25.020943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.020911    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:25.045835 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.045801    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"]
Nov 01 06:36:25.051738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.051710    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0394fa4-q5gt4"]
Nov 01 06:36:25.449308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.449282    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"]
Nov 01 06:36:25.450234 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:36:25.450211    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode262958f_fc5e_4b5d_b231_775478f412b8.slice/crio-1f7e9fc7232dd10de0c24d4a26e7ae8ca72d43bbcad652fda85f8d384da5f77c WatchSource:0}: Error finding container 1f7e9fc7232dd10de0c24d4a26e7ae8ca72d43bbcad652fda85f8d384da5f77c: Status 404 returned error can't find the container with id 1f7e9fc7232dd10de0c24d4a26e7ae8ca72d43bbcad652fda85f8d384da5f77c
Nov 01 06:36:25.452578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.452544    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:36:25.452701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.452607    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:36:25.452701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.452654    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:36:25.552925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.552889    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" event={"ID":"e262958f-fc5e-4b5d-b231-775478f412b8","Type":"ContainerStarted","Data":"15b110063d99ec22033317a5ba213972b2a5ae5b1d40e0279d9930cd38493cf8"}
Nov 01 06:36:25.553107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.552951    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" event={"ID":"e262958f-fc5e-4b5d-b231-775478f412b8","Type":"ContainerStarted","Data":"1f7e9fc7232dd10de0c24d4a26e7ae8ca72d43bbcad652fda85f8d384da5f77c"}
Nov 01 06:36:25.668713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:25.668659    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podStartSLOduration=1.668645259 podStartE2EDuration="1.668645259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:25.666963041 +0000 UTC m=+1597.234769872" watchObservedRunningTime="2026-11-01 06:36:25.668645259 +0000 UTC m=+1597.236452089"
Nov 01 06:36:26.021558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:26.021526    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:26.026941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:26.026917    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:26.559794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:26.559762    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:26.560833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:26.560815    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:36:26.589568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:26.589544    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6900cf21-eafa-436b-85c0-fdca5918d958" path="/var/lib/kubelet/pods/6900cf21-eafa-436b-85c0-fdca5918d958/volumes"
Nov 01 06:36:32.948345 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:36:32.948307    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:36:42.045419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.045378    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"]
Nov 01 06:36:42.094696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.094664    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab73428-nn9l5"]
Nov 01 06:36:42.594144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.594102    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96d328a-eefe-4560-987c-dbb75f304a31" path="/var/lib/kubelet/pods/d96d328a-eefe-4560-987c-dbb75f304a31/volumes"
Nov 01 06:36:42.676964 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.676875    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"]
Nov 01 06:36:42.680887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.680869    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.743414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.743377    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.743688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.743666    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jb6x\" (UniqueName: \"kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.743927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.743888    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.783401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.783357    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:36:42.783401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.783378    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:36:42.783633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.783412    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:36:42.783633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.783457    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:36:42.786507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.786486    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"]
Nov 01 06:36:42.845199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.845165    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.845416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.845242    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.845416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.845276    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jb6x\" (UniqueName: \"kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.846054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.845963    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.847703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.847682    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.901536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.901506    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jb6x\" (UniqueName: \"kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x\") pod \"maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:42.994721 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:42.994640    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:43.368410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:43.368385    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"]
Nov 01 06:36:43.370084 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:36:43.370057    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504806e3_7540_4eb4_81fa_18fcaa39488a.slice/crio-9dd88837afb8170e31fd017c7d60a52d9fb8fb4375918510ce9caa9b0bed0af8 WatchSource:0}: Error finding container 9dd88837afb8170e31fd017c7d60a52d9fb8fb4375918510ce9caa9b0bed0af8: Status 404 returned error can't find the container with id 9dd88837afb8170e31fd017c7d60a52d9fb8fb4375918510ce9caa9b0bed0af8
Nov 01 06:36:43.686305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:43.686220    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2" event={"ID":"504806e3-7540-4eb4-81fa-18fcaa39488a","Type":"ContainerStarted","Data":"04d26139fba60d7e5fa39187730047430cc853ff4148381f0f41ce456a8a3d02"}
Nov 01 06:36:43.686305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:43.686271    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2" event={"ID":"504806e3-7540-4eb4-81fa-18fcaa39488a","Type":"ContainerStarted","Data":"9dd88837afb8170e31fd017c7d60a52d9fb8fb4375918510ce9caa9b0bed0af8"}
Nov 01 06:36:43.799959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:43.799906    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2" podStartSLOduration=1.7998900839999998 podStartE2EDuration="1.799890084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:43.796860309 +0000 UTC m=+1615.364667140" watchObservedRunningTime="2026-11-01 06:36:43.799890084 +0000 UTC m=+1615.367696958"
Nov 01 06:36:44.694621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:44.694587    4468 generic.go:356] "Generic (PLEG): container finished" podID="504806e3-7540-4eb4-81fa-18fcaa39488a" containerID="04d26139fba60d7e5fa39187730047430cc853ff4148381f0f41ce456a8a3d02" exitCode=0
Nov 01 06:36:44.695012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:44.694659    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2" event={"ID":"504806e3-7540-4eb4-81fa-18fcaa39488a","Type":"ContainerDied","Data":"04d26139fba60d7e5fa39187730047430cc853ff4148381f0f41ce456a8a3d02"}
Nov 01 06:36:45.858725 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.858691    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:45.886030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.886003    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x\" (UniqueName: \"kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x\") pod \"504806e3-7540-4eb4-81fa-18fcaa39488a\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") "
Nov 01 06:36:45.886165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.886049    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca\") pod \"504806e3-7540-4eb4-81fa-18fcaa39488a\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") "
Nov 01 06:36:45.886205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.886160    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca\") pod \"504806e3-7540-4eb4-81fa-18fcaa39488a\" (UID: \"504806e3-7540-4eb4-81fa-18fcaa39488a\") "
Nov 01 06:36:45.886406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.886385    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca" pod "504806e3-7540-4eb4-81fa-18fcaa39488a" (UID: "504806e3-7540-4eb4-81fa-18fcaa39488a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:36:45.886670 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.886655    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:45.888320 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.888297    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca" pod "504806e3-7540-4eb4-81fa-18fcaa39488a" (UID: "504806e3-7540-4eb4-81fa-18fcaa39488a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:36:45.888410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.888369    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x" pod "504806e3-7540-4eb4-81fa-18fcaa39488a" (UID: "504806e3-7540-4eb4-81fa-18fcaa39488a"). InnerVolumeSpecName "kube-api-access-4jb6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:36:45.987527 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.987419    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jb6x\" (UniqueName: \"kubernetes.io/projected/504806e3-7540-4eb4-81fa-18fcaa39488a-kube-api-access-4jb6x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:45.987527 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:45.987492    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504806e3-7540-4eb4-81fa-18fcaa39488a-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:46.708845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:46.708812    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"
Nov 01 06:36:46.708845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:46.708823    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2" event={"ID":"504806e3-7540-4eb4-81fa-18fcaa39488a","Type":"ContainerDied","Data":"9dd88837afb8170e31fd017c7d60a52d9fb8fb4375918510ce9caa9b0bed0af8"}
Nov 01 06:36:46.709096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:46.708869    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dd88837afb8170e31fd017c7d60a52d9fb8fb4375918510ce9caa9b0bed0af8"
Nov 01 06:36:50.577536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:50.577505    4468 scope.go:122] "RemoveContainer" containerID="2e54d98202f89e0d31838adfeb687c51a10596567b6125db9120c626c742daea"
Nov 01 06:36:50.597263 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:50.597241    4468 scope.go:122] "RemoveContainer" containerID="f9cbc8fc5ee262269d5a7c8c7a6e7760ed01ee74648917ea09acfa27373f1907"
Nov 01 06:36:50.607623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:36:50.607598    4468 scope.go:122] "RemoveContainer" containerID="e1236503c9ab053d50cd90030977d52bce1f5ad812afaa25ddefde62931ea1b0"
Nov 01 06:37:02.949213 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:37:02.949171    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:37:06.242836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:06.242799    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:37:15.044406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:15.044370    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"]
Nov 01 06:37:15.093506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:15.093457    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ac68-w2nlf"]
Nov 01 06:37:16.591150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:16.591110    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5145af-8c4f-41f2-9e58-e93f3a0fc12d" path="/var/lib/kubelet/pods/0e5145af-8c4f-41f2-9e58-e93f3a0fc12d/volumes"
Nov 01 06:37:22.729131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:22.729089    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"]
Nov 01 06:37:22.776637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:22.776595    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-df0a0131-b7fj2"]
Nov 01 06:37:24.590695 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:24.590664    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504806e3-7540-4eb4-81fa-18fcaa39488a" path="/var/lib/kubelet/pods/504806e3-7540-4eb4-81fa-18fcaa39488a/volumes"
Nov 01 06:37:31.166828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:31.166744    4468 generic.go:356] "Generic (PLEG): container finished" podID="e262958f-fc5e-4b5d-b231-775478f412b8" containerID="15b110063d99ec22033317a5ba213972b2a5ae5b1d40e0279d9930cd38493cf8" exitCode=0
Nov 01 06:37:31.166828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:31.166809    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" event={"ID":"e262958f-fc5e-4b5d-b231-775478f412b8","Type":"ContainerDied","Data":"15b110063d99ec22033317a5ba213972b2a5ae5b1d40e0279d9930cd38493cf8"}
Nov 01 06:37:31.167259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:31.167239    4468 scope.go:122] "RemoveContainer" containerID="15b110063d99ec22033317a5ba213972b2a5ae5b1d40e0279d9930cd38493cf8"
Nov 01 06:37:31.168988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:31.168959    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:31.169098 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:31.169083    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:31.169143 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:31.169122    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:37:32.175362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:32.175331    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" event={"ID":"e262958f-fc5e-4b5d-b231-775478f412b8","Type":"ContainerStarted","Data":"86f084dd07c92977d5d6b9c98fa4764298028432831a574c83f1e0a58292b554"}
Nov 01 06:37:32.962162 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:37:32.962125    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:37:33.033490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:33.026894    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:37:33.033490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:33.027021    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 06:37:33.033490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:33.027065    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 06:37:34.025037 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:34.024998    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 06:37:34.025809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:34.025776    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 06:37:35.024490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:35.023923    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 06:37:35.024490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:35.024002    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 06:37:36.023491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:36.022914    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 06:37:36.023491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:36.022975    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 06:37:37.021945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:37.021903    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 01 06:37:37.022250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:37.022220    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 01 06:37:38.026681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:38.026648    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:37:38.027180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:38.027071    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:37:38.027909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:38.027887    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 06:37:39.433098 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:39.433059    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:37:45.346570 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:45.346536    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"]
Nov 01 06:37:45.347202 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:45.346791    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy" containerID="cri-o://a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6" gracePeriod=30
Nov 01 06:37:45.396886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:45.396853    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"]
Nov 01 06:37:45.397397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:45.397345    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy" containerID="cri-o://0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1" gracePeriod=30
Nov 01 06:37:45.400257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:45.400234    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"]
Nov 01 06:37:45.400643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:45.400456    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy" containerID="cri-o://0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8" gracePeriod=30
Nov 01 06:37:50.664366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.663897    4468 scope.go:122] "RemoveContainer" containerID="bcc31e710ba238282b924c0b3877e931fe43224814678f557572ea48cc49652f"
Nov 01 06:37:50.690860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.690838    4468 scope.go:122] "RemoveContainer" containerID="8a0ff1becc4fca14542fb32f0f332fec36fc6209c58c5901f8e3f552609aaf1c"
Nov 01 06:37:50.693591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.693562    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:37:50.703872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.703748    4468 scope.go:122] "RemoveContainer" containerID="029dd20bee9d86ce338ac75c8d4ec69cdb2606247656a3c54de687cc85d0288e"
Nov 01 06:37:50.722547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.722519    4468 scope.go:122] "RemoveContainer" containerID="3152f0e9e9344b82d259c093e60b2fa6189c9331cbd2c48e7fbb6507378a72d5"
Nov 01 06:37:50.733575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.733438    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:37:50.733575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.733481    4468 scope.go:122] "RemoveContainer" containerID="4daa711d74dd81b282b743c2b3a979a874bfa71ed474f4931f6c0ea58e64b725"
Nov 01 06:37:50.735961 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.735943    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:37:50.788331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788301    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788441    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788510    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788543    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788588    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788628    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788674    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788720    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788798    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket\") pod \"1571f43b-fc62-4861-b609-16c62203a1f1\" (UID: \"1571f43b-fc62-4861-b609-16c62203a1f1\") "
Nov 01 06:37:50.788945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788835    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.789047 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.788977    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.789098 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789071    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.789188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789169    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.789249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789206    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:37:50.789504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789448    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.789504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789505    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.789504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789521    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.789504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789531    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1571f43b-fc62-4861-b609-16c62203a1f1-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.789504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.789540    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.791182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.791154    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:50.791256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.791198    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.791256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.791219    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:37:50.791924 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.791901    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9" pod "1571f43b-fc62-4861-b609-16c62203a1f1" (UID: "1571f43b-fc62-4861-b609-16c62203a1f1"). InnerVolumeSpecName "kube-api-access-9cdm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:50.890367 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890335    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.890626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890385    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.890626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890426    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.890626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890451    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.890626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890529    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.890626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890589    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.890626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890624    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890661    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890689    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890747    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890777    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890834    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890885    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890900    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.891027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.890967    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.891434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891105    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.891434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891115    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.891434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891137    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.891434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891204    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data\") pod \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\" (UID: \"44adfc8e-6de3-4f88-a7bc-23b3042a2320\") "
Nov 01 06:37:50.891434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891235    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.891434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891269    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data\") pod \"aadba30e-2774-4d82-ac07-efbf65db4acf\" (UID: \"aadba30e-2774-4d82-ac07-efbf65db4acf\") "
Nov 01 06:37:50.891910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891880    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.891910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891906    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdm9\" (UniqueName: \"kubernetes.io/projected/1571f43b-fc62-4861-b609-16c62203a1f1-kube-api-access-9cdm9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.892120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891921    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.892120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891937    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1571f43b-fc62-4861-b609-16c62203a1f1-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.892120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891950    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.892120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.891967    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1571f43b-fc62-4861-b609-16c62203a1f1-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.892327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.892146    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.892387 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.892341    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:37:50.892866 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.892830    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.893027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.892987    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.893122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.893050    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.893267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.893248    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.893368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.893266    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.894223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.894172    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.894223 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.894197    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:50.894392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.894371    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "kube-api-access-dlm8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:50.894911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.894883    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:50.895590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.895566    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "kube-api-access-v4zw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:50.895699 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.895681    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:37:50.896187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.896163    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo" pod "aadba30e-2774-4d82-ac07-efbf65db4acf" (UID: "aadba30e-2774-4d82-ac07-efbf65db4acf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:37:50.896673 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.896655    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo" pod "44adfc8e-6de3-4f88-a7bc-23b3042a2320" (UID: "44adfc8e-6de3-4f88-a7bc-23b3042a2320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:37:50.994919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994820    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.994919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994853    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.994919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994867    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.994919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994880    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.994919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994892    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.994919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994916    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994928    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994941    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994953    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zw2\" (UniqueName: \"kubernetes.io/projected/44adfc8e-6de3-4f88-a7bc-23b3042a2320-kube-api-access-v4zw2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994965    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994978    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadba30e-2774-4d82-ac07-efbf65db4acf-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.994988    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.995000    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44adfc8e-6de3-4f88-a7bc-23b3042a2320-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.995011    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.995022    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadba30e-2774-4d82-ac07-efbf65db4acf-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:50.995256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:50.995034    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlm8c\" (UniqueName: \"kubernetes.io/projected/aadba30e-2774-4d82-ac07-efbf65db4acf-kube-api-access-dlm8c\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:51.315701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.315662    4468 generic.go:356] "Generic (PLEG): container finished" podID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerID="0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1" exitCode=0
Nov 01 06:37:51.315900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.315730    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"
Nov 01 06:37:51.315900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.315736    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" event={"ID":"aadba30e-2774-4d82-ac07-efbf65db4acf","Type":"ContainerDied","Data":"0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1"}
Nov 01 06:37:51.315900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.315811    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh" event={"ID":"aadba30e-2774-4d82-ac07-efbf65db4acf","Type":"ContainerDied","Data":"1cc64c0143884508846219ce5c19da6b458ce2b8e418a58f578052a63345dab5"}
Nov 01 06:37:51.315900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.315847    4468 scope.go:122] "RemoveContainer" containerID="0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1"
Nov 01 06:37:51.317669 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.317641    4468 generic.go:356] "Generic (PLEG): container finished" podID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerID="0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8" exitCode=0
Nov 01 06:37:51.317797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.317705    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"
Nov 01 06:37:51.317797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.317730    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" event={"ID":"44adfc8e-6de3-4f88-a7bc-23b3042a2320","Type":"ContainerDied","Data":"0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8"}
Nov 01 06:37:51.317797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.317792    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd" event={"ID":"44adfc8e-6de3-4f88-a7bc-23b3042a2320","Type":"ContainerDied","Data":"c0dc649b1952763e93cdbbb98cf14cbb574ed7b33302273f876c63b35f190ea7"}
Nov 01 06:37:51.319374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.319355    4468 generic.go:356] "Generic (PLEG): container finished" podID="1571f43b-fc62-4861-b609-16c62203a1f1" containerID="a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6" exitCode=0
Nov 01 06:37:51.319503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.319422    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"
Nov 01 06:37:51.319503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.319441    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" event={"ID":"1571f43b-fc62-4861-b609-16c62203a1f1","Type":"ContainerDied","Data":"a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6"}
Nov 01 06:37:51.319604 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.319533    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm" event={"ID":"1571f43b-fc62-4861-b609-16c62203a1f1","Type":"ContainerDied","Data":"6a1e213a58edd7efbe1aba1d113745e1d703adc6a76936aadc36586fae5c5387"}
Nov 01 06:37:51.328595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.328574    4468 scope.go:122] "RemoveContainer" containerID="0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1"
Nov 01 06:37:51.328917 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:37:51.328897    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1\": container with ID starting with 0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1 not found: ID does not exist" containerID="0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1"
Nov 01 06:37:51.329003 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.328930    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1"} err="failed to get container status \"0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1\": rpc error: code = NotFound desc = could not find container \"0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1\": container with ID starting with 0ce7e44cb18c453d4a2037453354b1b43a1b740657343c84b31470decd772bd1 not found: ID does not exist"
Nov 01 06:37:51.329003 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.328955    4468 scope.go:122] "RemoveContainer" containerID="0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8"
Nov 01 06:37:51.339186 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.339164    4468 scope.go:122] "RemoveContainer" containerID="0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8"
Nov 01 06:37:51.339624 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:37:51.339601    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8\": container with ID starting with 0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8 not found: ID does not exist" containerID="0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8"
Nov 01 06:37:51.339687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.339635    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8"} err="failed to get container status \"0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8\": rpc error: code = NotFound desc = could not find container \"0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8\": container with ID starting with 0a217cff1072682824acb5e1134486d447effcbe5526059b5168e1579dc5dce8 not found: ID does not exist"
Nov 01 06:37:51.339687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.339655    4468 scope.go:122] "RemoveContainer" containerID="a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6"
Nov 01 06:37:51.349596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.349572    4468 scope.go:122] "RemoveContainer" containerID="a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6"
Nov 01 06:37:51.349979 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:37:51.349959    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6\": container with ID starting with a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6 not found: ID does not exist" containerID="a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6"
Nov 01 06:37:51.350036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.349989    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6"} err="failed to get container status \"a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6\": rpc error: code = NotFound desc = could not find container \"a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6\": container with ID starting with a2ba107131d7b4ed1f207fe8fa002ce902cb8366b13ca6f25cfe655858ec8ec6 not found: ID does not exist"
Nov 01 06:37:51.480564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.480530    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"]
Nov 01 06:37:51.483195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.483172    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-r8jnh"]
Nov 01 06:37:51.590242 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.590143    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"]
Nov 01 06:37:51.593835 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.593808    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-4gqwm"]
Nov 01 06:37:51.765794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.765753    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"]
Nov 01 06:37:51.769302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:51.769275    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-vfwmd"]
Nov 01 06:37:52.590769 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:52.590736    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" path="/var/lib/kubelet/pods/1571f43b-fc62-4861-b609-16c62203a1f1/volumes"
Nov 01 06:37:52.591220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:52.591207    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" path="/var/lib/kubelet/pods/44adfc8e-6de3-4f88-a7bc-23b3042a2320/volumes"
Nov 01 06:37:52.591631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:37:52.591617    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" path="/var/lib/kubelet/pods/aadba30e-2774-4d82-ac07-efbf65db4acf/volumes"
Nov 01 06:38:02.947682 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:38:02.947640    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:38:06.267998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.267963    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"]
Nov 01 06:38:06.268536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268521    4468 cpu_manager.go:414] "containerMap: removing container" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268536    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268554    4468 cpu_manager.go:414] "containerMap: removing container" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268559    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268573    4468 cpu_manager.go:414] "containerMap: removing container" podUID="504806e3-7540-4eb4-81fa-18fcaa39488a" containerName="revoke-keys"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268579    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504806e3-7540-4eb4-81fa-18fcaa39488a" containerName="revoke-keys"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268593    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy"
Nov 01 06:38:06.268596 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268598    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy"
Nov 01 06:38:06.268809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268604    4468 cpu_manager.go:414] "containerMap: removing container" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy"
Nov 01 06:38:06.268809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268608    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy"
Nov 01 06:38:06.268809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268693    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadba30e-2774-4d82-ac07-efbf65db4acf" containerName="istio-proxy"
Nov 01 06:38:06.268809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268703    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="504806e3-7540-4eb4-81fa-18fcaa39488a" containerName="revoke-keys"
Nov 01 06:38:06.268809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268710    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1571f43b-fc62-4861-b609-16c62203a1f1" containerName="istio-proxy"
Nov 01 06:38:06.268809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.268715    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy"
Nov 01 06:38:06.273855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.273838    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.326026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.325993    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-51aa8f-openshift-default-dockercfg-cp677\""
Nov 01 06:38:06.380294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.380248    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"]
Nov 01 06:38:06.390785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.390752    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.390998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.390926    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.390998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.390971    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwg4\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.391149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.391054    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.391149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.391121    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.391267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.391189    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.391267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.391219    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.391267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.391265    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.391417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.391368    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.502596    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.503389    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.503577    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.503619    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwg4\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.504223    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.504341    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.504412    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.504671    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.504721    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.505863    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.504596    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.506861    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.506927    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.522556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.507148    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.545493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.544072    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.548888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.548815    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.570153 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.570101    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwg4\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.574521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.574410    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token\") pod \"e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") " pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:06.592299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:06.591910    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:07.051546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.051511    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"]
Nov 01 06:38:07.052327 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:38:07.052302    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef0a7a7_9749_464c_8726_74d5b5210eb7.slice/crio-95132dbe05dfe6837234e9c20e73393787cfef8118cef7cd0f43be72c82c8114 WatchSource:0}: Error finding container 95132dbe05dfe6837234e9c20e73393787cfef8118cef7cd0f43be72c82c8114: Status 404 returned error can't find the container with id 95132dbe05dfe6837234e9c20e73393787cfef8118cef7cd0f43be72c82c8114
Nov 01 06:38:07.054934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.054901    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:38:07.055010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.054968    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:38:07.055051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.055019    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:38:07.422886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.422798    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" event={"ID":"1ef0a7a7-9749-464c-8726-74d5b5210eb7","Type":"ContainerStarted","Data":"8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380"}
Nov 01 06:38:07.422886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.422859    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" event={"ID":"1ef0a7a7-9749-464c-8726-74d5b5210eb7","Type":"ContainerStarted","Data":"95132dbe05dfe6837234e9c20e73393787cfef8118cef7cd0f43be72c82c8114"}
Nov 01 06:38:07.539139 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.539073    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podStartSLOduration=1.539056317 podStartE2EDuration="1.539056317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:07.536731177 +0000 UTC m=+1699.104538033" watchObservedRunningTime="2026-11-01 06:38:07.539056317 +0000 UTC m=+1699.106863220"
Nov 01 06:38:07.593275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:07.593242    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:08.599237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:08.599206    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:09.437730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:09.437698    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:09.438743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:09.438721    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:38:13.922260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:13.922215    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:38:32.950212 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:38:32.950161    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:38:44.727908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:44.727874    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:38:50.772410 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:38:50.772379    4468 scope.go:122] "RemoveContainer" containerID="a11dae01161079fc8a28eb67f6b7c22ac442c6f588fcce120f7a2f0b0164c296"
Nov 01 06:39:02.948672 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:39:02.948588    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:39:10.810580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.810538    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"]
Nov 01 06:39:10.811507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.811450    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="44adfc8e-6de3-4f88-a7bc-23b3042a2320" containerName="istio-proxy"
Nov 01 06:39:10.814741 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.814725    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:10.870898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.870867    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"]
Nov 01 06:39:10.876658 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.876634    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:39:10.878987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.878555    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:39:10.878987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.878694    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:39:10.878987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.878730    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:39:10.912551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.912513    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:10.912743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.912717    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:10.912919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:10.912889    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82pj\" (UniqueName: \"kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.039490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.026681    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82pj\" (UniqueName: \"kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.039490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.026842    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.039490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.026953    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.039490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.027859    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.039490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.030949    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.083928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.083842    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82pj\" (UniqueName: \"kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj\") pod \"maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.128432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.128405    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:11.234443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.234409    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:39:11.502246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.502219    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"]
Nov 01 06:39:11.503589 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:39:11.503563    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae79ab5a_ea3e_4d26_9724_9bc231204015.slice/crio-5ab62069be2710ee800ac3d3d1df0d72878c73b42a6e1baea180ca328c1ed5cc WatchSource:0}: Error finding container 5ab62069be2710ee800ac3d3d1df0d72878c73b42a6e1baea180ca328c1ed5cc: Status 404 returned error can't find the container with id 5ab62069be2710ee800ac3d3d1df0d72878c73b42a6e1baea180ca328c1ed5cc
Nov 01 06:39:11.830698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.830664    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n" event={"ID":"ae79ab5a-ea3e-4d26-9724-9bc231204015","Type":"ContainerStarted","Data":"cacddbb88bd3f210696163637f1ead4fe12fab5c4433574b5b22ec9115d89b4c"}
Nov 01 06:39:11.831109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.830714    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n" event={"ID":"ae79ab5a-ea3e-4d26-9724-9bc231204015","Type":"ContainerStarted","Data":"5ab62069be2710ee800ac3d3d1df0d72878c73b42a6e1baea180ca328c1ed5cc"}
Nov 01 06:39:11.945750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:11.945693    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n" podStartSLOduration=1.9456791930000001 podStartE2EDuration="1.945679193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:39:11.942386508 +0000 UTC m=+1763.510193334" watchObservedRunningTime="2026-11-01 06:39:11.945679193 +0000 UTC m=+1763.513486023"
Nov 01 06:39:12.839049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:12.839014    4468 generic.go:356] "Generic (PLEG): container finished" podID="ae79ab5a-ea3e-4d26-9724-9bc231204015" containerID="cacddbb88bd3f210696163637f1ead4fe12fab5c4433574b5b22ec9115d89b4c" exitCode=0
Nov 01 06:39:12.839515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:12.839113    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n" event={"ID":"ae79ab5a-ea3e-4d26-9724-9bc231204015","Type":"ContainerDied","Data":"cacddbb88bd3f210696163637f1ead4fe12fab5c4433574b5b22ec9115d89b4c"}
Nov 01 06:39:14.056327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.056304    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:14.068784 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.068757    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj\" (UniqueName: \"kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj\") pod \"ae79ab5a-ea3e-4d26-9724-9bc231204015\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") "
Nov 01 06:39:14.068902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.068803    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca\") pod \"ae79ab5a-ea3e-4d26-9724-9bc231204015\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") "
Nov 01 06:39:14.068902 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.068842    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca\") pod \"ae79ab5a-ea3e-4d26-9724-9bc231204015\" (UID: \"ae79ab5a-ea3e-4d26-9724-9bc231204015\") "
Nov 01 06:39:14.069167 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.069144    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca" pod "ae79ab5a-ea3e-4d26-9724-9bc231204015" (UID: "ae79ab5a-ea3e-4d26-9724-9bc231204015"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:39:14.069267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.069244    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:14.070990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.070960    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca" pod "ae79ab5a-ea3e-4d26-9724-9bc231204015" (UID: "ae79ab5a-ea3e-4d26-9724-9bc231204015"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:39:14.071107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.071091    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj" pod "ae79ab5a-ea3e-4d26-9724-9bc231204015" (UID: "ae79ab5a-ea3e-4d26-9724-9bc231204015"). InnerVolumeSpecName "kube-api-access-g82pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:39:14.170890 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.170741    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae79ab5a-ea3e-4d26-9724-9bc231204015-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:14.170890 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.170773    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82pj\" (UniqueName: \"kubernetes.io/projected/ae79ab5a-ea3e-4d26-9724-9bc231204015-kube-api-access-g82pj\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:14.852012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.851929    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"
Nov 01 06:39:14.852184 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.851928    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n" event={"ID":"ae79ab5a-ea3e-4d26-9724-9bc231204015","Type":"ContainerDied","Data":"5ab62069be2710ee800ac3d3d1df0d72878c73b42a6e1baea180ca328c1ed5cc"}
Nov 01 06:39:14.852184 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:14.852057    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ab62069be2710ee800ac3d3d1df0d72878c73b42a6e1baea180ca328c1ed5cc"
Nov 01 06:39:17.045271 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:17.045235    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"]
Nov 01 06:39:17.094489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:17.094422    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3123b6-pxp59"]
Nov 01 06:39:18.589913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:18.589878    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6" path="/var/lib/kubelet/pods/d8bcb0d1-4e56-4e78-9d67-fb8e9a7bcfb6/volumes"
Nov 01 06:39:21.897429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:21.897393    4468 generic.go:356] "Generic (PLEG): container finished" podID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerID="8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380" exitCode=0
Nov 01 06:39:21.897987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:21.897487    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" event={"ID":"1ef0a7a7-9749-464c-8726-74d5b5210eb7","Type":"ContainerDied","Data":"8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380"}
Nov 01 06:39:21.898061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:21.897996    4468 scope.go:122] "RemoveContainer" containerID="8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380"
Nov 01 06:39:21.899439 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:21.899409    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:39:21.899521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:21.899508    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:39:21.899560 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:21.899543    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:39:22.929987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:22.929950    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" event={"ID":"1ef0a7a7-9749-464c-8726-74d5b5210eb7","Type":"ContainerStarted","Data":"c35d4acbd12322a22d62e94142f995a38ecae63163ed2bbc4b64f2a95da34237"}
Nov 01 06:39:23.593374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:23.593339    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:39:23.594688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:23.594655    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 06:39:23.594811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:23.594709    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 06:39:24.437796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:24.437758    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:39:24.593341 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:24.593308    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 06:39:24.593539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:24.593377    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 06:39:25.593456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:25.593423    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 06:39:25.593874 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:25.593513    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 06:39:26.592974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:26.592943    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 06:39:26.593141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:26.593009    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 06:39:27.592870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:27.592836    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 06:39:27.593359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:27.592887    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 06:39:28.597767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:28.597737    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:39:28.598838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:28.598818    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 06:39:32.947230 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:39:32.947194    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:39:50.045668 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:50.045634    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"]
Nov 01 06:39:50.095129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:50.095100    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e84b520-h6jmz"]
Nov 01 06:39:50.590403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:50.590371    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18caf57d-919f-48b9-be20-de7e9e5139d4" path="/var/lib/kubelet/pods/18caf57d-919f-48b9-be20-de7e9e5139d4/volumes"
Nov 01 06:39:58.771164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:58.771130    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"]
Nov 01 06:39:58.820290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:39:58.820264    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8b4338ad-qz78n"]
Nov 01 06:40:00.590604 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:00.590567    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae79ab5a-ea3e-4d26-9724-9bc231204015" path="/var/lib/kubelet/pods/ae79ab5a-ea3e-4d26-9724-9bc231204015/volumes"
Nov 01 06:40:02.947478 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:02.947425    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:40:08.124234 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:08.124196    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"]
Nov 01 06:40:08.124742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:08.124388    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy" containerID="cri-o://cdbb8909375599f6fde13ce79414aa89ebd66a06cbfe50746dfe66ddc9cb8e52" gracePeriod=30
Nov 01 06:40:08.174193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:08.174163    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"]
Nov 01 06:40:08.174441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:08.174417    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy" containerID="cri-o://9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f" gracePeriod=30
Nov 01 06:40:13.260158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.260130    4468 generic.go:356] "Generic (PLEG): container finished" podID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerID="cdbb8909375599f6fde13ce79414aa89ebd66a06cbfe50746dfe66ddc9cb8e52" exitCode=0
Nov 01 06:40:13.260505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.260214    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" event={"ID":"91b84a1c-36e2-44c7-9074-289cbc3e2698","Type":"ContainerDied","Data":"cdbb8909375599f6fde13ce79414aa89ebd66a06cbfe50746dfe66ddc9cb8e52"}
Nov 01 06:40:13.439531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.439505    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:40:13.465752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.465731    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:40:13.562353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562320    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.562528 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562404    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562528 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562444    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.562661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562542    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.562661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562572    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562600    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562643    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562679    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562740    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562791    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.562869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562828    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.563072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562876    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.563072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562964    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.563072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562993    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.563072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563026    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.563072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562639    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.563072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563058    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562670    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.562829    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563030    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563098    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563297    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket\") pod \"8b983509-8fc0-4982-8991-7ec621e59169\" (UID: \"8b983509-8fc0-4982-8991-7ec621e59169\") "
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563393    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs\") pod \"91b84a1c-36e2-44c7-9074-289cbc3e2698\" (UID: \"91b84a1c-36e2-44c7-9074-289cbc3e2698\") "
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563584    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563888    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563898    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563919    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563933    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563949    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b983509-8fc0-4982-8991-7ec621e59169-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563962    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.564016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.563978    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.564774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.564504    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.564774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.564615    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.564943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.564809    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.565603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.565577    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.565984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.565963    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:13.566264 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.566221    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:13.566378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.566351    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:13.566841 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.566817    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:13.566956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.566917    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "kube-api-access-xt6b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:13.567033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.567015    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8" pod "91b84a1c-36e2-44c7-9074-289cbc3e2698" (UID: "91b84a1c-36e2-44c7-9074-289cbc3e2698"). InnerVolumeSpecName "kube-api-access-wtnn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:13.567173 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.567160    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo" pod "8b983509-8fc0-4982-8991-7ec621e59169" (UID: "8b983509-8fc0-4982-8991-7ec621e59169"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:13.665675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665586    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665614    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665629    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665644    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b983509-8fc0-4982-8991-7ec621e59169-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665656    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665669    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt6b4\" (UniqueName: \"kubernetes.io/projected/8b983509-8fc0-4982-8991-7ec621e59169-kube-api-access-xt6b4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665682    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665696    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b84a1c-36e2-44c7-9074-289cbc3e2698-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665708    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b84a1c-36e2-44c7-9074-289cbc3e2698-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665719    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b84a1c-36e2-44c7-9074-289cbc3e2698-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665730    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnn8\" (UniqueName: \"kubernetes.io/projected/91b84a1c-36e2-44c7-9074-289cbc3e2698-kube-api-access-wtnn8\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:13.665992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:13.665742    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b983509-8fc0-4982-8991-7ec621e59169-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:14.267860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.267826    4468 generic.go:356] "Generic (PLEG): container finished" podID="8b983509-8fc0-4982-8991-7ec621e59169" containerID="9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f" exitCode=0
Nov 01 06:40:14.268280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.267878    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"
Nov 01 06:40:14.268280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.267926    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" event={"ID":"8b983509-8fc0-4982-8991-7ec621e59169","Type":"ContainerDied","Data":"9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f"}
Nov 01 06:40:14.268280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.267976    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj" event={"ID":"8b983509-8fc0-4982-8991-7ec621e59169","Type":"ContainerDied","Data":"de141bc1da74cc636e43729b8f7c5b4ae1a896b6b1454c0a6075820001587d45"}
Nov 01 06:40:14.268280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.268005    4468 scope.go:122] "RemoveContainer" containerID="9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f"
Nov 01 06:40:14.269530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.269507    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k" event={"ID":"91b84a1c-36e2-44c7-9074-289cbc3e2698","Type":"ContainerDied","Data":"63ff8de268b938207c483b7c8f38ac5db18a33ccd47020e48a6f2c7e5c95445f"}
Nov 01 06:40:14.269814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.269539    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"
Nov 01 06:40:14.282777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.282757    4468 scope.go:122] "RemoveContainer" containerID="9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f"
Nov 01 06:40:14.283259 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:14.283233    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f\": container with ID starting with 9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f not found: ID does not exist" containerID="9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f"
Nov 01 06:40:14.283362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.283271    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f"} err="failed to get container status \"9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f\": rpc error: code = NotFound desc = could not find container \"9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f\": container with ID starting with 9616e52195dd1e97cd5c4c8a2514e6c5dc4669bfad5f51dbea12ee7a9dcf926f not found: ID does not exist"
Nov 01 06:40:14.283362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.283292    4468 scope.go:122] "RemoveContainer" containerID="cdbb8909375599f6fde13ce79414aa89ebd66a06cbfe50746dfe66ddc9cb8e52"
Nov 01 06:40:14.433119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.433039    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"]
Nov 01 06:40:14.436943 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.436916    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fxl5k"]
Nov 01 06:40:14.544531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.544502    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"]
Nov 01 06:40:14.550045 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.550012    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-n2vdj"]
Nov 01 06:40:14.590103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.590077    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b983509-8fc0-4982-8991-7ec621e59169" path="/var/lib/kubelet/pods/8b983509-8fc0-4982-8991-7ec621e59169/volumes"
Nov 01 06:40:14.590555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:14.590541    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" path="/var/lib/kubelet/pods/91b84a1c-36e2-44c7-9074-289cbc3e2698/volumes"
Nov 01 06:40:18.954382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:18.954347    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:40:31.475321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:31.475234    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"]
Nov 01 06:40:31.475875 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:31.475575    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" containerID="cri-o://1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290" gracePeriod=30
Nov 01 06:40:32.398067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:32.398027    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"]
Nov 01 06:40:32.398275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:32.398237    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" containerID="cri-o://36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80" gracePeriod=30
Nov 01 06:40:32.948245 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:32.948206    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:40:33.245638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:33.245570    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:40:33.245770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:33.245640    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:40:36.740526 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.740498    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:40:36.752404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.752354    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:40:36.752543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.752412    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:40:36.832623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832594    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.832756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832632    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.832756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832691    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.832756 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832737    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.832917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832778    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.832917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832824    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:36.832917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832832    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.833077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832963    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:36.833077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.832996    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.833077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833058    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.833241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833098    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy\") pod \"fb219425-e174-40b6-b762-678a4eb173b1\" (UID: \"fb219425-e174-40b6-b762-678a4eb173b1\") "
Nov 01 06:40:36.833241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833056    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:36.833241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833069    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:36.833241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833135    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:36.833861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833834    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.833994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833865    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.833994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833882    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.833994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833897    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.833994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.833911    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb219425-e174-40b6-b762-678a4eb173b1-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.835051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.835026    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:36.835161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.835144    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:36.835222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.835199    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "kube-api-access-vh6rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:36.835290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.835277    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy" pod "fb219425-e174-40b6-b762-678a4eb173b1" (UID: "fb219425-e174-40b6-b762-678a4eb173b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:36.935277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.935251    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.935277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.935277    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6rt\" (UniqueName: \"kubernetes.io/projected/fb219425-e174-40b6-b762-678a4eb173b1-kube-api-access-vh6rt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.935493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.935287    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb219425-e174-40b6-b762-678a4eb173b1-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:36.935493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:36.935297    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb219425-e174-40b6-b762-678a4eb173b1-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.421491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.421450    4468 generic.go:356] "Generic (PLEG): container finished" podID="fb219425-e174-40b6-b762-678a4eb173b1" containerID="1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290" exitCode=0
Nov 01 06:40:37.421603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.421495    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" event={"ID":"fb219425-e174-40b6-b762-678a4eb173b1","Type":"ContainerDied","Data":"1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290"}
Nov 01 06:40:37.421603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.421530    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"
Nov 01 06:40:37.421603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.421551    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97" event={"ID":"fb219425-e174-40b6-b762-678a4eb173b1","Type":"ContainerDied","Data":"73f85db794cbff5349bf39c559f972c357c195baebc4a8ff2a5c350c231c0d78"}
Nov 01 06:40:37.421603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.421573    4468 scope.go:122] "RemoveContainer" containerID="1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290"
Nov 01 06:40:37.511385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.511367    4468 scope.go:122] "RemoveContainer" containerID="1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290"
Nov 01 06:40:37.511794 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:37.511702    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290\": container with ID starting with 1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290 not found: ID does not exist" containerID="1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290"
Nov 01 06:40:37.511794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.511732    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290"} err="failed to get container status \"1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290\": rpc error: code = NotFound desc = could not find container \"1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290\": container with ID starting with 1f4ea657833ab0215a5fd569cc2347e1daf84d7b3b9afedeb402698baa141290 not found: ID does not exist"
Nov 01 06:40:37.585036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.585007    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"]
Nov 01 06:40:37.633702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.633675    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-6dt97"]
Nov 01 06:40:37.654027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.653999    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:40:37.745330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745267    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745363    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745427    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745495    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745549    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745594    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745646    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.745738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745704    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.746100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745771    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token\") pod \"7274ce5d-f935-4730-94ef-1fda35773de2\" (UID: \"7274ce5d-f935-4730-94ef-1fda35773de2\") "
Nov 01 06:40:37.746100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745774    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:37.746100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745817    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:37.746100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745834    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:37.746100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.745891    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:37.746100 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.746078    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:37.746452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.746433    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.746574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.746460    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.746574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.746504    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.746574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.746536    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.746574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.746550    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7274ce5d-f935-4730-94ef-1fda35773de2-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.747712 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.747694    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "kube-api-access-n88sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:37.747872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.747857    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:37.748392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.748375    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:37.748451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.748414    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token" pod "7274ce5d-f935-4730-94ef-1fda35773de2" (UID: "7274ce5d-f935-4730-94ef-1fda35773de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:37.847832 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.847799    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.847832 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.847829    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88sx\" (UniqueName: \"kubernetes.io/projected/7274ce5d-f935-4730-94ef-1fda35773de2-kube-api-access-n88sx\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.848030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.847844    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7274ce5d-f935-4730-94ef-1fda35773de2-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:37.848030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:37.847856    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7274ce5d-f935-4730-94ef-1fda35773de2-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:38.428831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.428799    4468 generic.go:356] "Generic (PLEG): container finished" podID="7274ce5d-f935-4730-94ef-1fda35773de2" containerID="36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80" exitCode=0
Nov 01 06:40:38.429015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.428848    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"
Nov 01 06:40:38.429015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.428877    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" event={"ID":"7274ce5d-f935-4730-94ef-1fda35773de2","Type":"ContainerDied","Data":"36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80"}
Nov 01 06:40:38.429015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.428934    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f" event={"ID":"7274ce5d-f935-4730-94ef-1fda35773de2","Type":"ContainerDied","Data":"aff4411082bf9c143b4fa5a9ae70a55fb60f9173ab023c45bd5877cdad5da685"}
Nov 01 06:40:38.429015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.428956    4468 scope.go:122] "RemoveContainer" containerID="36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80"
Nov 01 06:40:38.439931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.439915    4468 scope.go:122] "RemoveContainer" containerID="36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80"
Nov 01 06:40:38.440184 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:38.440168    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80\": container with ID starting with 36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80 not found: ID does not exist" containerID="36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80"
Nov 01 06:40:38.440244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.440191    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80"} err="failed to get container status \"36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80\": rpc error: code = NotFound desc = could not find container \"36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80\": container with ID starting with 36eb8dcff3ce68bfccc0e45903165215a57ccf3c52cc3b727f09f5ca9fda5f80 not found: ID does not exist"
Nov 01 06:40:38.590620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.590580    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb219425-e174-40b6-b762-678a4eb173b1" path="/var/lib/kubelet/pods/fb219425-e174-40b6-b762-678a4eb173b1/volumes"
Nov 01 06:40:38.592979 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.592954    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"]
Nov 01 06:40:38.642331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:38.642304    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-j958f"]
Nov 01 06:40:40.590830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:40.590795    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" path="/var/lib/kubelet/pods/7274ce5d-f935-4730-94ef-1fda35773de2/volumes"
Nov 01 06:40:46.594433 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:46.594404    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"]
Nov 01 06:40:46.594934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:46.594643    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" podUID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" containerName="discovery" containerID="cri-o://b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27" gracePeriod=30
Nov 01 06:40:46.869229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:46.869205    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:40:47.050653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050622    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.050849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050671    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.050849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050755    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.050849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050783    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.050849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050817    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.051082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050890    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.051082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.050968    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token\") pod \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\" (UID: \"288cfc7d-4091-4bef-949b-5b6ee8900f8b\") "
Nov 01 06:40:47.051182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.051143    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:47.051688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.051659    4468 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-ca-configmap\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.053664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.053639    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:40:47.053772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.053678    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:47.053965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.053927    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:47.054033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.054008    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "kube-api-access-czjsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:47.054270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.054253    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:40:47.054330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.054315    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts" pod "288cfc7d-4091-4bef-949b-5b6ee8900f8b" (UID: "288cfc7d-4091-4bef-949b-5b6ee8900f8b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:40:47.153105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.153009    4468 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-csr-dns-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.153105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.153048    4468 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-kubeconfig\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.153105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.153061    4468 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/288cfc7d-4091-4bef-949b-5b6ee8900f8b-cacerts\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.153105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.153077    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.153105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.153089    4468 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cfc7d-4091-4bef-949b-5b6ee8900f8b-local-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.153105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.153101    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czjsh\" (UniqueName: \"kubernetes.io/projected/288cfc7d-4091-4bef-949b-5b6ee8900f8b-kube-api-access-czjsh\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:47.484346 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.484260    4468 generic.go:356] "Generic (PLEG): container finished" podID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" containerID="b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27" exitCode=0
Nov 01 06:40:47.484524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.484383    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"
Nov 01 06:40:47.484524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.484408    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" event={"ID":"288cfc7d-4091-4bef-949b-5b6ee8900f8b","Type":"ContainerDied","Data":"b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27"}
Nov 01 06:40:47.484524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.484500    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh" event={"ID":"288cfc7d-4091-4bef-949b-5b6ee8900f8b","Type":"ContainerDied","Data":"114d22f0363fb162ec50747ae38126525c68a425c3c71732ba2eebf2fee67693"}
Nov 01 06:40:47.484651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.484527    4468 scope.go:122] "RemoveContainer" containerID="b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27"
Nov 01 06:40:47.495921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.495888    4468 scope.go:122] "RemoveContainer" containerID="b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27"
Nov 01 06:40:47.496246 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:47.496225    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27\": container with ID starting with b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27 not found: ID does not exist" containerID="b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27"
Nov 01 06:40:47.496371 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.496253    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27"} err="failed to get container status \"b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27\": rpc error: code = NotFound desc = could not find container \"b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27\": container with ID starting with b718c5045abdf38b3306b91dbf319e5b9e325ca80bab9329da5b21a89e598f27 not found: ID does not exist"
Nov 01 06:40:47.500976 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.500954    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"]
Nov 01 06:40:47.501166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.501146    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" containerID="cri-o://19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319" gracePeriod=30
Nov 01 06:40:47.770817 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.770734    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"]
Nov 01 06:40:47.774415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:47.774392    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7glh"]
Nov 01 06:40:48.590312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:48.590281    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" path="/var/lib/kubelet/pods/288cfc7d-4091-4bef-949b-5b6ee8900f8b/volumes"
Nov 01 06:40:49.761055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:49.761015    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:40:49.761453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:49.761091    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:40:50.817747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:50.817716    4468 scope.go:122] "RemoveContainer" containerID="cb50ed9a13a85f37f60ded3ebc392262a6da53bb96a1ea3a671dce0dba5fc66c"
Nov 01 06:40:50.827321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:50.827301    4468 scope.go:122] "RemoveContainer" containerID="347f7394ee654f7e4695c9b74436e456881e20c967161c7e3ff8c56bd7cbdfe2"
Nov 01 06:40:50.835728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:50.835706    4468 scope.go:122] "RemoveContainer" containerID="94ff860ea220b6d7c004284cffb609257d567c38f6a833d8206f0c0da9204796"
Nov 01 06:40:52.761045 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.761023    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:40:52.811126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811100    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811137    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811186    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811214    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811283    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811360    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811407    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811439    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811502    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs\") pod \"7e777d1c-f8aa-43e3-9263-670757d9239f\" (UID: \"7e777d1c-f8aa-43e3-9263-670757d9239f\") "
Nov 01 06:40:52.811610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811585    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:52.811772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811582    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:52.811830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811779    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:52.811896 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.811803    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:52.812162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.812011    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:52.812162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.812172    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.812162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.812190    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.812162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.812206    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e777d1c-f8aa-43e3-9263-670757d9239f-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.812162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.812220    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.813549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.813524    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:52.813661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.813590    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:52.813845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.813824    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "kube-api-access-lp9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:52.813845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.813834    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy" pod "7e777d1c-f8aa-43e3-9263-670757d9239f" (UID: "7e777d1c-f8aa-43e3-9263-670757d9239f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:52.913207 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.913176    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.913207 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.913205    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9jp\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-kube-api-access-lp9jp\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.913390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.913215    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.913390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.913224    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e777d1c-f8aa-43e3-9263-670757d9239f-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:52.913390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:52.913232    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e777d1c-f8aa-43e3-9263-670757d9239f-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:53.519248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.519216    4468 generic.go:356] "Generic (PLEG): container finished" podID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerID="19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319" exitCode=0
Nov 01 06:40:53.519414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.519264    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"
Nov 01 06:40:53.519414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.519273    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" event={"ID":"7e777d1c-f8aa-43e3-9263-670757d9239f","Type":"ContainerDied","Data":"19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319"}
Nov 01 06:40:53.519414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.519316    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz" event={"ID":"7e777d1c-f8aa-43e3-9263-670757d9239f","Type":"ContainerDied","Data":"f702cec9255f1199582b8f8f5734df8b0d0d9be9f21060e4b5a44cf8003ae0ff"}
Nov 01 06:40:53.519414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.519335    4468 scope.go:122] "RemoveContainer" containerID="19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319"
Nov 01 06:40:53.529450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.529425    4468 scope.go:122] "RemoveContainer" containerID="19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319"
Nov 01 06:40:53.529748 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:40:53.529728    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319\": container with ID starting with 19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319 not found: ID does not exist" containerID="19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319"
Nov 01 06:40:53.529805 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.529756    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319"} err="failed to get container status \"19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319\": rpc error: code = NotFound desc = could not find container \"19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319\": container with ID starting with 19ea64b76dc790ee4baa7da239a9540c5c5f25b277a8c0452a209aa589a21319 not found: ID does not exist"
Nov 01 06:40:53.731838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.731803    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"]
Nov 01 06:40:53.734972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:53.734951    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-cffsz"]
Nov 01 06:40:54.590421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:54.590366    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" path="/var/lib/kubelet/pods/7e777d1c-f8aa-43e3-9263-670757d9239f/volumes"
Nov 01 06:40:59.840386 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840346    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"]
Nov 01 06:40:59.840857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840842    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy"
Nov 01 06:40:59.840905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840858    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy"
Nov 01 06:40:59.840905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840874    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ae79ab5a-ea3e-4d26-9724-9bc231204015" containerName="revoke-keys"
Nov 01 06:40:59.840905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840879    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae79ab5a-ea3e-4d26-9724-9bc231204015" containerName="revoke-keys"
Nov 01 06:40:59.840905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840892    4468 cpu_manager.go:414] "containerMap: removing container" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy"
Nov 01 06:40:59.840905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840897    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840913    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840917    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840923    4468 cpu_manager.go:414] "containerMap: removing container" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840928    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840937    4468 cpu_manager.go:414] "containerMap: removing container" podUID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" containerName="discovery"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840941    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" containerName="discovery"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840953    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.840959    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841042    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b84a1c-36e2-44c7-9074-289cbc3e2698" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841054    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7274ce5d-f935-4730-94ef-1fda35773de2" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841060    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb219425-e174-40b6-b762-678a4eb173b1" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841066    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="288cfc7d-4091-4bef-949b-5b6ee8900f8b" containerName="discovery"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841074    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b983509-8fc0-4982-8991-7ec621e59169" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841081    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e777d1c-f8aa-43e3-9263-670757d9239f" containerName="istio-proxy"
Nov 01 06:40:59.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.841088    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae79ab5a-ea3e-4d26-9724-9bc231204015" containerName="revoke-keys"
Nov 01 06:40:59.845785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.845766    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.884645 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.884604    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.884814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.884748    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xrg\" (UniqueName: \"kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.884814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.884798    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.884935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.884846    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.885023 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.885002    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.885084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.885057    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.901685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.901656    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"]
Nov 01 06:40:59.947814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.947784    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-51aa8f\"/\"kube-root-ca.crt\""
Nov 01 06:40:59.947814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.947809    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-51aa8f\"/\"default-dockercfg-lgx69\""
Nov 01 06:40:59.947814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.947784    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-51aa8f\"/\"openshift-service-ca.crt\""
Nov 01 06:40:59.948049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.947834    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-51aa8f\"/\"auth-test-model-51aa8f-kserve-self-signed-certs\""
Nov 01 06:40:59.987391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.987348    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xrg\" (UniqueName: \"kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.987409    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.987551    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.987620    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.987658    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.987854    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.988054    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.988200    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.987913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.988431    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.991189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.990929    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:40:59.991189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:40:59.991145    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:00.044198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:00.044161    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xrg\" (UniqueName: \"kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg\") pod \"auth-test-model-51aa8f-kserve-67f6f4b485-tn46p\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") " pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:00.158090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:00.157992    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:00.532379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:00.532353    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"]
Nov 01 06:41:00.534068 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:41:00.534042    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59cdcf27_c4f1_4679_a8ec_d2e9ea73da47.slice/crio-b3658d1149c51b946229f08435ad38cd9c2835480fd3e2575a31cbf4bc83e5b4 WatchSource:0}: Error finding container b3658d1149c51b946229f08435ad38cd9c2835480fd3e2575a31cbf4bc83e5b4: Status 404 returned error can't find the container with id b3658d1149c51b946229f08435ad38cd9c2835480fd3e2575a31cbf4bc83e5b4
Nov 01 06:41:00.559342 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:00.559310    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" event={"ID":"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47","Type":"ContainerStarted","Data":"b3658d1149c51b946229f08435ad38cd9c2835480fd3e2575a31cbf4bc83e5b4"}
Nov 01 06:41:01.568222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:01.568187    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" event={"ID":"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47","Type":"ContainerStarted","Data":"522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e"}
Nov 01 06:41:01.681663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:01.681606    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" podStartSLOduration=2.681592775 podStartE2EDuration="2.681592775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:41:01.680486963 +0000 UTC m=+1873.248293792" watchObservedRunningTime="2026-11-01 06:41:01.681592775 +0000 UTC m=+1873.249399605"
Nov 01 06:41:02.158116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:02.158074    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:02.240764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:02.240731    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:41:02.948604 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:41:02.948572    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:41:10.158948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:10.158909    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:10.172146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:10.172115    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:10.629124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:10.629095    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:41:23.139432 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:23.139394    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:41:28.887313 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.887268    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"]
Nov 01 06:41:28.891372 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.891348    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:28.939409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.939376    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:28.939648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.939550    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:28.939708 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.939668    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:28.939860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.939838    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbxxx\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:28.945564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.945533    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-97c3302b7889\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.994933    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"kube-root-ca.crt\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.994951    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"routing-overlay\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.994951    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ff1dc\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.994967    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-529628bc0fc6\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.995009    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-e2e--528f7081-dockercfg-c6b6l\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.994986    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-841de3b011bd\""
Nov 01 06:41:28.995052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.995071    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"openshift-service-ca.crt\""
Nov 01 06:41:28.997544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:28.997520    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"]
Nov 01 06:41:29.040739 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.040687    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.040951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.040801    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.040951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.040831    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.040951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.040925    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbxxx\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.041459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.041434    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.041615 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.041597    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.043665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.043648    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.097679 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.097645    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbxxx\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.206089 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.205994    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.585009 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.584982    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"]
Nov 01 06:41:29.586694 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:41:29.586662    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c49fec_c528_4ddc_8604_945abd7e4b08.slice/crio-20bfff7b308e0b34ab7a858c71d58ec128e7a0284097c1f253a65522aae427c1 WatchSource:0}: Error finding container 20bfff7b308e0b34ab7a858c71d58ec128e7a0284097c1f253a65522aae427c1: Status 404 returned error can't find the container with id 20bfff7b308e0b34ab7a858c71d58ec128e7a0284097c1f253a65522aae427c1
Nov 01 06:41:29.757835 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.757793    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9" event={"ID":"c5c49fec-c528-4ddc-8604-945abd7e4b08","Type":"ContainerStarted","Data":"d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641"}
Nov 01 06:41:29.758054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.757849    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9" event={"ID":"c5c49fec-c528-4ddc-8604-945abd7e4b08","Type":"ContainerStarted","Data":"20bfff7b308e0b34ab7a858c71d58ec128e7a0284097c1f253a65522aae427c1"}
Nov 01 06:41:29.758054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.757873    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:29.872998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:29.872892    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9" podStartSLOduration=1.872878022 podStartE2EDuration="1.872878022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:41:29.870178265 +0000 UTC m=+1901.437985119" watchObservedRunningTime="2026-11-01 06:41:29.872878022 +0000 UTC m=+1901.440684852"
Nov 01 06:41:31.380008 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.379970    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"]
Nov 01 06:41:31.383933 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.383912    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.436389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.436355    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"kube-root-ca.crt\""
Nov 01 06:41:31.436389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.436367    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-d41a0e39961b\""
Nov 01 06:41:31.471823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.471776    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.471995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.471927    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.471995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.471967    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzwj\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.471995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.471990    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.485927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.485895    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25719d\""
Nov 01 06:41:31.486166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.485926    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-f6f2772a70c5\""
Nov 01 06:41:31.486166 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.485934    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"routing-overlay\""
Nov 01 06:41:31.486345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.485932    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"openshift-service-ca.crt\""
Nov 01 06:41:31.486345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.485970    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-294b20a2cf15\""
Nov 01 06:41:31.486345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.485984    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-e2e--bf6d8971-dockercfg-6qmzh\""
Nov 01 06:41:31.489809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.489783    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"]
Nov 01 06:41:31.573526 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.573447    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.573758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.573556    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzwj\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.573758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.573581    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.573758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.573716    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.574420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.574352    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.574420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.574371    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.576502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.576455    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.629935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.629897    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzwj\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:31.698118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:31.698027    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:32.072109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:32.072076    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"]
Nov 01 06:41:32.073546 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:41:32.073498    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c29ceab_c25c_4650_b5ed_6f85d5244c33.slice/crio-2ba0b19f08da70d72511b9a37a41658cc627c0f2ca10bacacf9a529e161d7833 WatchSource:0}: Error finding container 2ba0b19f08da70d72511b9a37a41658cc627c0f2ca10bacacf9a529e161d7833: Status 404 returned error can't find the container with id 2ba0b19f08da70d72511b9a37a41658cc627c0f2ca10bacacf9a529e161d7833
Nov 01 06:41:32.780593 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:32.780555    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2" event={"ID":"0c29ceab-c25c-4650-b5ed-6f85d5244c33","Type":"ContainerStarted","Data":"a097a132eac1de3fc4f9f6f50446d44b3229b7b65f5bcc7d8cfddc4cef10d797"}
Nov 01 06:41:32.781111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:32.780614    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2" event={"ID":"0c29ceab-c25c-4650-b5ed-6f85d5244c33","Type":"ContainerStarted","Data":"2ba0b19f08da70d72511b9a37a41658cc627c0f2ca10bacacf9a529e161d7833"}
Nov 01 06:41:32.781111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:32.780642    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:32.894203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:32.894148    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2" podStartSLOduration=1.894132895 podStartE2EDuration="1.894132895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:41:32.892418135 +0000 UTC m=+1904.460224964" watchObservedRunningTime="2026-11-01 06:41:32.894132895 +0000 UTC m=+1904.461939759"
Nov 01 06:41:32.948943 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:41:32.948903    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:41:36.918682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:36.918637    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"]
Nov 01 06:41:36.955855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:36.955828    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:36.975995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:36.975961    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"]
Nov 01 06:41:37.008511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.008447    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-4e5348\"/\"kube-root-ca.crt\""
Nov 01 06:41:37.021656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.021629    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-4e5348\"/\"openshift-service-ca.crt\""
Nov 01 06:41:37.055444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.055414    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls\") pod \"e2e-usage-sim-bf7ae6-58db9f9888-l6p8w\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") " pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.055611 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.055592    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5txv\" (UniqueName: \"kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv\") pod \"e2e-usage-sim-bf7ae6-58db9f9888-l6p8w\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") " pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.058214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.058194    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-4e5348\"/\"default-dockercfg-489kb\""
Nov 01 06:41:37.058291 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.058225    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-4e5348\"/\"e2e-usage-sim-bf7ae6-cert\""
Nov 01 06:41:37.156554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.156517    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls\") pod \"e2e-usage-sim-bf7ae6-58db9f9888-l6p8w\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") " pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.156744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.156638    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5txv\" (UniqueName: \"kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv\") pod \"e2e-usage-sim-bf7ae6-58db9f9888-l6p8w\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") " pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.159340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.159315    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls\") pod \"e2e-usage-sim-bf7ae6-58db9f9888-l6p8w\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") " pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.213116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.213047    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5txv\" (UniqueName: \"kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv\") pod \"e2e-usage-sim-bf7ae6-58db9f9888-l6p8w\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") " pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.274684 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.274650    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.670624 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.670599    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"]
Nov 01 06:41:37.672168 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:41:37.672141    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddb5aa83_7a4c_41f5_be5a_e4c67a52b791.slice/crio-086ad1c885fd23711d5d25c45ba59f695c28b26586101261a62df89262ae7e76 WatchSource:0}: Error finding container 086ad1c885fd23711d5d25c45ba59f695c28b26586101261a62df89262ae7e76: Status 404 returned error can't find the container with id 086ad1c885fd23711d5d25c45ba59f695c28b26586101261a62df89262ae7e76
Nov 01 06:41:37.810937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.810903    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w" event={"ID":"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791","Type":"ContainerStarted","Data":"e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b"}
Nov 01 06:41:37.811113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.810948    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w" event={"ID":"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791","Type":"ContainerStarted","Data":"086ad1c885fd23711d5d25c45ba59f695c28b26586101261a62df89262ae7e76"}
Nov 01 06:41:37.811113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.810973    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.816256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.816229    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:41:37.926363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:37.926255    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w" podStartSLOduration=1.9262405120000001 podStartE2EDuration="1.926240512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:41:37.925059755 +0000 UTC m=+1909.492866596" watchObservedRunningTime="2026-11-01 06:41:37.926240512 +0000 UTC m=+1909.494047342"
Nov 01 06:41:40.766907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:40.766875    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:41:43.790629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:43.790598    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:41:57.640858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:41:57.640824    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:42:02.947617 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:42:02.947525    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:42:03.042216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.042168    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"]
Nov 01 06:42:03.047411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.047387    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.102436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.102404    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"]
Nov 01 06:42:03.148612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.148575    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:42:03.148782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.148657    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:42:03.149238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.149222    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:42:03.149279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.149248    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:42:03.167478 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.167442    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.167675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.167616    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwvc\" (UniqueName: \"kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.167786 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.167768    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.269063 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.268970    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.269063 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.269038    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwvc\" (UniqueName: \"kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.269306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.269110    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.269678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.269647    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.271725 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.271699    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.325645 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.325599    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwvc\" (UniqueName: \"kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc\") pod \"maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.360527 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.360458    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:03.747582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.747548    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"]
Nov 01 06:42:03.748303 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:42:03.748277    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4522d41_24ee_4fcb_94f2_ec1c22db4f80.slice/crio-188d54cb6532bc6ab138ae9a0f175abb2de5102c5511b0f9773337bfc8fa3b1d WatchSource:0}: Error finding container 188d54cb6532bc6ab138ae9a0f175abb2de5102c5511b0f9773337bfc8fa3b1d: Status 404 returned error can't find the container with id 188d54cb6532bc6ab138ae9a0f175abb2de5102c5511b0f9773337bfc8fa3b1d
Nov 01 06:42:03.960989 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:03.960952    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl" event={"ID":"c4522d41-24ee-4fcb-94f2-ec1c22db4f80","Type":"ContainerStarted","Data":"188d54cb6532bc6ab138ae9a0f175abb2de5102c5511b0f9773337bfc8fa3b1d"}
Nov 01 06:42:04.969803 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:04.969770    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl" event={"ID":"c4522d41-24ee-4fcb-94f2-ec1c22db4f80","Type":"ContainerStarted","Data":"93aac4bf4e26a4b310309cfacb89d6353cf18c21cf454675cd29f695b565dee6"}
Nov 01 06:42:05.083812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:05.083756    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl" podStartSLOduration=2.083741238 podStartE2EDuration="2.083741238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:05.081494241 +0000 UTC m=+1936.649301066" watchObservedRunningTime="2026-11-01 06:42:05.083741238 +0000 UTC m=+1936.651548068"
Nov 01 06:42:10.771956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.771921    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"]
Nov 01 06:42:10.775726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.775709    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.832366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.832326    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"]
Nov 01 06:42:10.861112 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.861076    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zgt\" (UniqueName: \"kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.861305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.861164    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.861382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.861314    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.962969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.962931    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.963141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.963028    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.963141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.963118    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zgt\" (UniqueName: \"kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.963757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.963734    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:10.965423 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:10.965391    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:11.020054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:11.020016    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zgt\" (UniqueName: \"kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt\") pod \"maas-api-revoke-sub-93b0eb1a-q88nv\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:11.089544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:11.089510    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:11.461828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:11.461802    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"]
Nov 01 06:42:11.463215 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:42:11.463169    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbce2f4e_6fda_439b_8b40_416cd700cb98.slice/crio-2cbbe5dd58fe4a48b9df19eb142c2704858094cd18ebf239b43f1b282cefeb71 WatchSource:0}: Error finding container 2cbbe5dd58fe4a48b9df19eb142c2704858094cd18ebf239b43f1b282cefeb71: Status 404 returned error can't find the container with id 2cbbe5dd58fe4a48b9df19eb142c2704858094cd18ebf239b43f1b282cefeb71
Nov 01 06:42:12.024801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:12.024698    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv" event={"ID":"dbce2f4e-6fda-439b-8b40-416cd700cb98","Type":"ContainerStarted","Data":"0a41fd9ad12b9c548889fdd81c517e141321294a7be5dba4bfae448f9724bb9e"}
Nov 01 06:42:12.024801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:12.024783    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv" event={"ID":"dbce2f4e-6fda-439b-8b40-416cd700cb98","Type":"ContainerStarted","Data":"2cbbe5dd58fe4a48b9df19eb142c2704858094cd18ebf239b43f1b282cefeb71"}
Nov 01 06:42:12.141277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:12.141198    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv" podStartSLOduration=2.141178239 podStartE2EDuration="2.141178239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:12.138812374 +0000 UTC m=+1943.706619204" watchObservedRunningTime="2026-11-01 06:42:12.141178239 +0000 UTC m=+1943.708985074"
Nov 01 06:42:13.032346 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:13.032308    4468 generic.go:356] "Generic (PLEG): container finished" podID="dbce2f4e-6fda-439b-8b40-416cd700cb98" containerID="0a41fd9ad12b9c548889fdd81c517e141321294a7be5dba4bfae448f9724bb9e" exitCode=0
Nov 01 06:42:13.032819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:13.032393    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv" event={"ID":"dbce2f4e-6fda-439b-8b40-416cd700cb98","Type":"ContainerDied","Data":"0a41fd9ad12b9c548889fdd81c517e141321294a7be5dba4bfae448f9724bb9e"}
Nov 01 06:42:14.189887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.189864    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:14.300691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.300664    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca\") pod \"dbce2f4e-6fda-439b-8b40-416cd700cb98\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") "
Nov 01 06:42:14.300854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.300772    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt\" (UniqueName: \"kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt\") pod \"dbce2f4e-6fda-439b-8b40-416cd700cb98\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") "
Nov 01 06:42:14.300917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.300897    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca\") pod \"dbce2f4e-6fda-439b-8b40-416cd700cb98\" (UID: \"dbce2f4e-6fda-439b-8b40-416cd700cb98\") "
Nov 01 06:42:14.301040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.301013    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca" pod "dbce2f4e-6fda-439b-8b40-416cd700cb98" (UID: "dbce2f4e-6fda-439b-8b40-416cd700cb98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:14.301279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.301258    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:14.302910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.302887    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt" pod "dbce2f4e-6fda-439b-8b40-416cd700cb98" (UID: "dbce2f4e-6fda-439b-8b40-416cd700cb98"). InnerVolumeSpecName "kube-api-access-p7zgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:14.303023 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.302956    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca" pod "dbce2f4e-6fda-439b-8b40-416cd700cb98" (UID: "dbce2f4e-6fda-439b-8b40-416cd700cb98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:42:14.402592 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.402565    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbce2f4e-6fda-439b-8b40-416cd700cb98-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:14.402592 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:14.402589    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7zgt\" (UniqueName: \"kubernetes.io/projected/dbce2f4e-6fda-439b-8b40-416cd700cb98-kube-api-access-p7zgt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:15.044382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:15.044299    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"
Nov 01 06:42:15.044382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:15.044325    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv" event={"ID":"dbce2f4e-6fda-439b-8b40-416cd700cb98","Type":"ContainerDied","Data":"2cbbe5dd58fe4a48b9df19eb142c2704858094cd18ebf239b43f1b282cefeb71"}
Nov 01 06:42:15.044382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:15.044374    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cbbe5dd58fe4a48b9df19eb142c2704858094cd18ebf239b43f1b282cefeb71"
Nov 01 06:42:32.947669 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:42:32.947633    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:42:35.169157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.169121    4468 generic.go:356] "Generic (PLEG): container finished" podID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerID="93aac4bf4e26a4b310309cfacb89d6353cf18c21cf454675cd29f695b565dee6" exitCode=28
Nov 01 06:42:35.169617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.169172    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl" event={"ID":"c4522d41-24ee-4fcb-94f2-ec1c22db4f80","Type":"ContainerDied","Data":"93aac4bf4e26a4b310309cfacb89d6353cf18c21cf454675cd29f695b565dee6"}
Nov 01 06:42:35.169822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.169803    4468 scope.go:122] "RemoveContainer" containerID="93aac4bf4e26a4b310309cfacb89d6353cf18c21cf454675cd29f695b565dee6"
Nov 01 06:42:35.348171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.348137    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"]
Nov 01 06:42:35.348706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.348691    4468 cpu_manager.go:414] "containerMap: removing container" podUID="dbce2f4e-6fda-439b-8b40-416cd700cb98" containerName="revoke-keys"
Nov 01 06:42:35.348706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.348708    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbce2f4e-6fda-439b-8b40-416cd700cb98" containerName="revoke-keys"
Nov 01 06:42:35.348827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.348815    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbce2f4e-6fda-439b-8b40-416cd700cb98" containerName="revoke-keys"
Nov 01 06:42:35.353777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.353753    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.407146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407119    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-1a1f3593a873\""
Nov 01 06:42:35.407402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407161    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-2b38f786e9be\""
Nov 01 06:42:35.407402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407189    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\""
Nov 01 06:42:35.407402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407219    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\""
Nov 01 06:42:35.407402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407311    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\""
Nov 01 06:42:35.407402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407322    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\""
Nov 01 06:42:35.407402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407376    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\""
Nov 01 06:42:35.407819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.407410    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\""
Nov 01 06:42:35.408496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.408446    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"]
Nov 01 06:42:35.450723 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.450645    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtg7\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.450861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.450795    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.450861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.450854    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.450966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.450950    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.507988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.507963    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\""
Nov 01 06:42:35.551864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.551837    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.551984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.551920    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtg7\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.552072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.552052    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.552133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.552116    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.552619 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.552597    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.552808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.552778    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.555074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.555052    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.608391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.608367    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtg7\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:35.670729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:35.669639    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:36.043284 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.043261    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"]
Nov 01 06:42:36.044821 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:42:36.044789    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7f04ad_5fda_4ed8_81d3_f8cc1235fd48.slice/crio-70eb5ed4a589ffb0a4091352e4a00ac6854dda2e7dd8c6f8c826da57acaa0639 WatchSource:0}: Error finding container 70eb5ed4a589ffb0a4091352e4a00ac6854dda2e7dd8c6f8c826da57acaa0639: Status 404 returned error can't find the container with id 70eb5ed4a589ffb0a4091352e4a00ac6854dda2e7dd8c6f8c826da57acaa0639
Nov 01 06:42:36.176965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.176926    4468 generic.go:356] "Generic (PLEG): container finished" podID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerID="889dd85e2af73334920da8c73c909e2e018ab3ff41a1120f76f27ea7f385d551" exitCode=0
Nov 01 06:42:36.177405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.177000    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl" event={"ID":"c4522d41-24ee-4fcb-94f2-ec1c22db4f80","Type":"ContainerDied","Data":"889dd85e2af73334920da8c73c909e2e018ab3ff41a1120f76f27ea7f385d551"}
Nov 01 06:42:36.177405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.177048    4468 scope.go:122] "RemoveContainer" containerID="93aac4bf4e26a4b310309cfacb89d6353cf18c21cf454675cd29f695b565dee6"
Nov 01 06:42:36.181392 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.181363    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" event={"ID":"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48","Type":"ContainerStarted","Data":"bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5"}
Nov 01 06:42:36.181547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.181415    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" event={"ID":"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48","Type":"ContainerStarted","Data":"70eb5ed4a589ffb0a4091352e4a00ac6854dda2e7dd8c6f8c826da57acaa0639"}
Nov 01 06:42:36.181547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.181447    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:36.404233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:36.404183    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" podStartSLOduration=1.404168436 podStartE2EDuration="1.404168436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:36.402363852 +0000 UTC m=+1967.970170683" watchObservedRunningTime="2026-11-01 06:42:36.404168436 +0000 UTC m=+1967.971975265"
Nov 01 06:42:37.335530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.335504    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:37.474864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.474793    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca\") pod \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") "
Nov 01 06:42:37.475020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.474950    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc\" (UniqueName: \"kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc\") pod \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") "
Nov 01 06:42:37.475020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.474988    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca\") pod \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\" (UID: \"c4522d41-24ee-4fcb-94f2-ec1c22db4f80\") "
Nov 01 06:42:37.475490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.475445    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca" pod "c4522d41-24ee-4fcb-94f2-ec1c22db4f80" (UID: "c4522d41-24ee-4fcb-94f2-ec1c22db4f80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:37.477451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.477429    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc" pod "c4522d41-24ee-4fcb-94f2-ec1c22db4f80" (UID: "c4522d41-24ee-4fcb-94f2-ec1c22db4f80"). InnerVolumeSpecName "kube-api-access-xlwvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:37.477612 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.477594    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca" pod "c4522d41-24ee-4fcb-94f2-ec1c22db4f80" (UID: "c4522d41-24ee-4fcb-94f2-ec1c22db4f80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:42:37.577064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.577035    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:37.577064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.577062    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwvc\" (UniqueName: \"kubernetes.io/projected/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-kube-api-access-xlwvc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:37.577258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:37.577072    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4522d41-24ee-4fcb-94f2-ec1c22db4f80-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:38.196533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:38.196498    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"
Nov 01 06:42:38.196533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:38.196522    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl" event={"ID":"c4522d41-24ee-4fcb-94f2-ec1c22db4f80","Type":"ContainerDied","Data":"188d54cb6532bc6ab138ae9a0f175abb2de5102c5511b0f9773337bfc8fa3b1d"}
Nov 01 06:42:38.196755 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:38.196573    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="188d54cb6532bc6ab138ae9a0f175abb2de5102c5511b0f9773337bfc8fa3b1d"
Nov 01 06:42:39.292171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.292134    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"]
Nov 01 06:42:39.292733 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.292716    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerName="revoke-keys"
Nov 01 06:42:39.292778 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.292734    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerName="revoke-keys"
Nov 01 06:42:39.292778 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.292755    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerName="revoke-keys"
Nov 01 06:42:39.292778 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.292760    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerName="revoke-keys"
Nov 01 06:42:39.292868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.292847    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerName="revoke-keys"
Nov 01 06:42:39.296282 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.296264    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.402575    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.402632    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.402676    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.402757    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.402814    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.402978    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgr9\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.403103    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.403196    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.403246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.403251    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.406548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.406523    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"]
Nov 01 06:42:39.504288 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504252    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504296    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504329    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504511    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504541    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504594    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgr9\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504615    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504634    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504684    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504748    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.504915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504899    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.505240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.504916    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.505240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.505025    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.505421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.505380    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.506746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.506723    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.507231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.507214    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.561110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.561080    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.561401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.561377    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgr9\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7jm7x\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:39.609330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:39.609303    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:40.038644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.038615    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"]
Nov 01 06:42:40.040037 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:42:40.040006    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769a8b93_1638_4313_887b_128fc0512de5.slice/crio-5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b WatchSource:0}: Error finding container 5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b: Status 404 returned error can't find the container with id 5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b
Nov 01 06:42:40.042434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.042401    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:42:40.042512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.042494    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:42:40.042554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.042526    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:42:40.210431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.210396    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" event={"ID":"769a8b93-1638-4313-887b-128fc0512de5","Type":"ContainerStarted","Data":"c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224"}
Nov 01 06:42:40.210676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.210459    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" event={"ID":"769a8b93-1638-4313-887b-128fc0512de5","Type":"ContainerStarted","Data":"5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b"}
Nov 01 06:42:40.327600 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.327538    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podStartSLOduration=1.327518756 podStartE2EDuration="1.327518756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:40.325116601 +0000 UTC m=+1971.892923432" watchObservedRunningTime="2026-11-01 06:42:40.327518756 +0000 UTC m=+1971.895325588"
Nov 01 06:42:40.609961 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:40.609879    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:41.627703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:41.627645    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:41.628236 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:41.627731    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:42.627516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:42.624913    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:42.627516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:42.624968    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:42.934195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:42.934158    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"]
Nov 01 06:42:43.040502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:43.040447    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"]
Nov 01 06:42:43.625493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:43.623752    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:43.625493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:43.623812    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:44.628747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:44.628645    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:44.628747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:44.628705    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:45.616493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:45.609784    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:45.616493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:45.609840    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:46.610286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:46.610249    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:46.610707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:46.610312    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:47.192998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:47.192969    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:42:47.610235 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:47.610200    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 01 06:42:47.610413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:47.610261    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 01 06:42:47.738008 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:47.737976    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:42:47.935048 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:47.934960    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9" podUID="c5c49fec-c528-4ddc-8604-945abd7e4b08" containerName="payload-processing" containerID="cri-o://d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641" gracePeriod=25
Nov 01 06:42:48.041897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.041843    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2" podUID="0c29ceab-c25c-4650-b5ed-6f85d5244c33" containerName="payload-processing" containerID="cri-o://a097a132eac1de3fc4f9f6f50446d44b3229b7b65f5bcc7d8cfddc4cef10d797" gracePeriod=25
Nov 01 06:42:48.281737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.281493    4468 generic.go:356] "Generic (PLEG): container finished" podID="0c29ceab-c25c-4650-b5ed-6f85d5244c33" containerID="a097a132eac1de3fc4f9f6f50446d44b3229b7b65f5bcc7d8cfddc4cef10d797" exitCode=0
Nov 01 06:42:48.281737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.281646    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2" event={"ID":"0c29ceab-c25c-4650-b5ed-6f85d5244c33","Type":"ContainerDied","Data":"a097a132eac1de3fc4f9f6f50446d44b3229b7b65f5bcc7d8cfddc4cef10d797"}
Nov 01 06:42:48.283883 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.283843    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:42:48.286765 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.285929    4468 generic.go:356] "Generic (PLEG): container finished" podID="c5c49fec-c528-4ddc-8604-945abd7e4b08" containerID="d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641" exitCode=0
Nov 01 06:42:48.286765 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.285968    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9" event={"ID":"c5c49fec-c528-4ddc-8604-945abd7e4b08","Type":"ContainerDied","Data":"d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641"}
Nov 01 06:42:48.286765 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.285998    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9" event={"ID":"c5c49fec-c528-4ddc-8604-945abd7e4b08","Type":"ContainerDied","Data":"20bfff7b308e0b34ab7a858c71d58ec128e7a0284097c1f253a65522aae427c1"}
Nov 01 06:42:48.286765 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.286018    4468 scope.go:122] "RemoveContainer" containerID="d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641"
Nov 01 06:42:48.306614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.301845    4468 scope.go:122] "RemoveContainer" containerID="d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641"
Nov 01 06:42:48.306614 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:42:48.302187    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641\": container with ID starting with d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641 not found: ID does not exist" containerID="d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641"
Nov 01 06:42:48.306614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.302222    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641"} err="failed to get container status \"d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641\": rpc error: code = NotFound desc = could not find container \"d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641\": container with ID starting with d174e20e57cf0b49bd9bcdae08d480181c3d9f7e30499cdfd70bd32d60a56641 not found: ID does not exist"
Nov 01 06:42:48.346498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.345592    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx\") pod \"c5c49fec-c528-4ddc-8604-945abd7e4b08\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") "
Nov 01 06:42:48.346498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.345666    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume\") pod \"c5c49fec-c528-4ddc-8604-945abd7e4b08\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") "
Nov 01 06:42:48.346498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.345710    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume\") pod \"c5c49fec-c528-4ddc-8604-945abd7e4b08\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") "
Nov 01 06:42:48.346498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.345868    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials\") pod \"c5c49fec-c528-4ddc-8604-945abd7e4b08\" (UID: \"c5c49fec-c528-4ddc-8604-945abd7e4b08\") "
Nov 01 06:42:48.348626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.347259    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume" pod "c5c49fec-c528-4ddc-8604-945abd7e4b08" (UID: "c5c49fec-c528-4ddc-8604-945abd7e4b08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:48.348970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.348923    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume" pod "c5c49fec-c528-4ddc-8604-945abd7e4b08" (UID: "c5c49fec-c528-4ddc-8604-945abd7e4b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:48.364006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.363608    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials" pod "c5c49fec-c528-4ddc-8604-945abd7e4b08" (UID: "c5c49fec-c528-4ddc-8604-945abd7e4b08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:48.365581 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.365545    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx" pod "c5c49fec-c528-4ddc-8604-945abd7e4b08" (UID: "c5c49fec-c528-4ddc-8604-945abd7e4b08"). InnerVolumeSpecName "kube-api-access-pbxxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:48.400811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.400784    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:42:48.447816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.447758    4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-provider-credentials\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.447816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.447781    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbxxx\" (UniqueName: \"kubernetes.io/projected/c5c49fec-c528-4ddc-8604-945abd7e4b08-kube-api-access-pbxxx\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.447816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.447791    4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-routing-overlay-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.447816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.447801    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c49fec-c528-4ddc-8604-945abd7e4b08-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.548704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.548669    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials\") pod \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") "
Nov 01 06:42:48.548992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.548972    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume\") pod \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") "
Nov 01 06:42:48.549696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.549569    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume" pod "0c29ceab-c25c-4650-b5ed-6f85d5244c33" (UID: "0c29ceab-c25c-4650-b5ed-6f85d5244c33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:48.549876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.549849    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume\") pod \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") "
Nov 01 06:42:48.550026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.550007    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj\") pod \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\" (UID: \"0c29ceab-c25c-4650-b5ed-6f85d5244c33\") "
Nov 01 06:42:48.550388 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.550364    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume" pod "0c29ceab-c25c-4650-b5ed-6f85d5244c33" (UID: "0c29ceab-c25c-4650-b5ed-6f85d5244c33"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:48.550651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.550629    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.550803 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.550780    4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c29ceab-c25c-4650-b5ed-6f85d5244c33-routing-overlay-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.552426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.552397    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials" pod "0c29ceab-c25c-4650-b5ed-6f85d5244c33" (UID: "0c29ceab-c25c-4650-b5ed-6f85d5244c33"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:48.552822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.552793    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj" pod "0c29ceab-c25c-4650-b5ed-6f85d5244c33" (UID: "0c29ceab-c25c-4650-b5ed-6f85d5244c33"). InnerVolumeSpecName "kube-api-access-gmzwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:48.652274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.652214    4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-provider-credentials\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.652709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.652249    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzwj\" (UniqueName: \"kubernetes.io/projected/0c29ceab-c25c-4650-b5ed-6f85d5244c33-kube-api-access-gmzwj\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:48.792087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.791976    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:48.792687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.792665    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:48.793823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:48.793804    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 06:42:49.295942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.295904    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2" event={"ID":"0c29ceab-c25c-4650-b5ed-6f85d5244c33","Type":"ContainerDied","Data":"2ba0b19f08da70d72511b9a37a41658cc627c0f2ca10bacacf9a529e161d7833"}
Nov 01 06:42:49.296203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.296187    4468 scope.go:122] "RemoveContainer" containerID="a097a132eac1de3fc4f9f6f50446d44b3229b7b65f5bcc7d8cfddc4cef10d797"
Nov 01 06:42:49.296355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.296340    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"
Nov 01 06:42:49.297831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.297808    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"
Nov 01 06:42:49.464306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.464270    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"]
Nov 01 06:42:49.467859 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.467831    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c87tkm2"]
Nov 01 06:42:49.577896 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.577809    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"]
Nov 01 06:42:49.581865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:49.581840    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95v5pd9"]
Nov 01 06:42:50.590554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:50.590520    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c29ceab-c25c-4650-b5ed-6f85d5244c33" path="/var/lib/kubelet/pods/0c29ceab-c25c-4650-b5ed-6f85d5244c33/volumes"
Nov 01 06:42:50.590994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:50.590977    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c49fec-c528-4ddc-8604-945abd7e4b08" path="/var/lib/kubelet/pods/c5c49fec-c528-4ddc-8604-945abd7e4b08/volumes"
Nov 01 06:42:50.900949 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:42:50.900878    4468 scope.go:122] "RemoveContainer" containerID="04d26139fba60d7e5fa39187730047430cc853ff4148381f0f41ce456a8a3d02"
Nov 01 06:43:02.947543 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:43:02.947504    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:43:09.521266 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.521230    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"]
Nov 01 06:43:09.521880 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.521863    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0c29ceab-c25c-4650-b5ed-6f85d5244c33" containerName="payload-processing"
Nov 01 06:43:09.521926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.521882    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c29ceab-c25c-4650-b5ed-6f85d5244c33" containerName="payload-processing"
Nov 01 06:43:09.521926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.521923    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c5c49fec-c528-4ddc-8604-945abd7e4b08" containerName="payload-processing"
Nov 01 06:43:09.521990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.521928    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c49fec-c528-4ddc-8604-945abd7e4b08" containerName="payload-processing"
Nov 01 06:43:09.522026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.522001    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c29ceab-c25c-4650-b5ed-6f85d5244c33" containerName="payload-processing"
Nov 01 06:43:09.522026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.522016    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c49fec-c528-4ddc-8604-945abd7e4b08" containerName="payload-processing"
Nov 01 06:43:09.522026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.522023    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" containerName="revoke-keys"
Nov 01 06:43:09.526904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.526884    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.618848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.618815    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dhf\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.618856    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.618876    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.619004    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.619039    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619213 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.619190    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619266 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.619249    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619413 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.619398    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.619508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.619460    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.631603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.631577    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"]
Nov 01 06:43:09.720286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720256    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720316    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dhf\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720337    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720355    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720400    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720433    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720486    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720521    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720622    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720701    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720791    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.720876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.720856    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.721125 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.721041    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.721125 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.721072    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.722924 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.722903    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.723139 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.723120    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.776957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.776886    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.777068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.777008    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dhf\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf\") pod \"maas-default-gateway-openshift-default-7d9897b54c-dwr2h\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:09.839994 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:09.839952    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:10.265991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.265966    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"]
Nov 01 06:43:10.267740 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:43:10.267718    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b88da6_fb37_4599_8683_90965863b521.slice/crio-f1f44025f753124d1a47e54cbeceff5622318c1c33fc3131e5e711d7ab8142d6 WatchSource:0}: Error finding container f1f44025f753124d1a47e54cbeceff5622318c1c33fc3131e5e711d7ab8142d6: Status 404 returned error can't find the container with id f1f44025f753124d1a47e54cbeceff5622318c1c33fc3131e5e711d7ab8142d6
Nov 01 06:43:10.270059 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.270029    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:43:10.270124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.270093    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:43:10.270160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.270121    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:43:10.468214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.468181    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" event={"ID":"a4b88da6-fb37-4599-8683-90965863b521","Type":"ContainerStarted","Data":"6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891"}
Nov 01 06:43:10.468403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.468229    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" event={"ID":"a4b88da6-fb37-4599-8683-90965863b521","Type":"ContainerStarted","Data":"f1f44025f753124d1a47e54cbeceff5622318c1c33fc3131e5e711d7ab8142d6"}
Nov 01 06:43:10.585280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.585230    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podStartSLOduration=1.585209402 podStartE2EDuration="1.585209402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:10.582089043 +0000 UTC m=+2002.149895896" watchObservedRunningTime="2026-11-01 06:43:10.585209402 +0000 UTC m=+2002.153016232"
Nov 01 06:43:10.841007 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:10.840927    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:11.842406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:11.842371    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:11.842851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:11.842426    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:12.840543 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:12.840501    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:12.840753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:12.840573    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:13.840742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:13.840708    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:13.841129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:13.840761    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:14.841064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:14.841024    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:14.841529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:14.841083    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:15.840417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:15.840362    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:15.840650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:15.840442    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:16.840379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:16.840344    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:16.840814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:16.840395    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:17.841295 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:17.841256    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 01 06:43:17.841804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:17.841317    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 01 06:43:18.922247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:18.922213    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:18.922967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:18.922932    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:18.923823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:18.923802    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 06:43:29.384119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:29.384040    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"]
Nov 01 06:43:29.433379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:29.433353    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c66acdd-lgjdl"]
Nov 01 06:43:30.590757 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:30.590720    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4522d41-24ee-4fcb-94f2-ec1c22db4f80" path="/var/lib/kubelet/pods/c4522d41-24ee-4fcb-94f2-ec1c22db4f80/volumes"
Nov 01 06:43:31.072738 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.072699    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"]
Nov 01 06:43:31.077090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.077071    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.130491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.130424    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-030936cd-openshift-default-dockercfg-kps8z\""
Nov 01 06:43:31.176684 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176654    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bkw\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.176882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176707    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.176882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176756    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.176882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176864    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.177057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176889    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.177057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176916    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.177057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.176985    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.177057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.177020    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.177256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.177071    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.189730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.189697    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"]
Nov 01 06:43:31.278257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278221    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278260    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278285    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278348    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278382    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278425    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278540    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bkw\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278585    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278641    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278820    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.278919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278841    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.279082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.278915    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.279200 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.279145    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.279315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.279174    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.280920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.280902    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.281033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.281016    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.334766 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.334714    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.335243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.335225    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bkw\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw\") pod \"e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") " pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.390481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.390451    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:31.815961 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.815936    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"]
Nov 01 06:43:31.817297 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:43:31.817272    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49078d4c_aabb_44cf_bea4_6b9a67de3f8b.slice/crio-9a00b41a04d1f6d754b75a7a63da01afbdd498f5254729ea83efbe7f213b9aac WatchSource:0}: Error finding container 9a00b41a04d1f6d754b75a7a63da01afbdd498f5254729ea83efbe7f213b9aac: Status 404 returned error can't find the container with id 9a00b41a04d1f6d754b75a7a63da01afbdd498f5254729ea83efbe7f213b9aac
Nov 01 06:43:31.819589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.819560    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:43:31.819653 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.819633    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:43:31.819689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:31.819664    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:43:32.598325 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:32.598292    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" event={"ID":"49078d4c-aabb-44cf-bea4-6b9a67de3f8b","Type":"ContainerStarted","Data":"7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"}
Nov 01 06:43:32.598539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:32.598343    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" event={"ID":"49078d4c-aabb-44cf-bea4-6b9a67de3f8b","Type":"ContainerStarted","Data":"9a00b41a04d1f6d754b75a7a63da01afbdd498f5254729ea83efbe7f213b9aac"}
Nov 01 06:43:32.765277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:32.765225    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podStartSLOduration=1.765211989 podStartE2EDuration="1.765211989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:32.762425196 +0000 UTC m=+2024.330232025" watchObservedRunningTime="2026-11-01 06:43:32.765211989 +0000 UTC m=+2024.333018876"
Nov 01 06:43:32.947694 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:43:32.947601    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:43:33.390878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:33.390846    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:33.396711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:33.396682    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:33.606008 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:33.605983    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:33.607044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:33.607023    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:43:35.785854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:35.785812    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:43:52.286716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.286675    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"]
Nov 01 06:43:52.290850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.290827    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.344912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.344886    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:43:52.391066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.391023    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.391228 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.391076    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npftd\" (UniqueName: \"kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.391324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.391302    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.394718 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.394699    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:43:52.394718 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.394714    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:43:52.394878 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.394854    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:43:52.397581 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.397565    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"]
Nov 01 06:43:52.492190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.492170    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.492319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.492207    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.492319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.492229    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npftd\" (UniqueName: \"kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.492945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.492922    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.494764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.494744    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.548623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.548597    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npftd\" (UniqueName: \"kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd\") pod \"maas-api-revoke-sub-93bfa728-5xfvc\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.605785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.605736    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:52.980204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:52.980177    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"]
Nov 01 06:43:52.981135 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:43:52.981111    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f55a17_d5f1_44f4_b24b_ebb04fff4f13.slice/crio-03b4ac250a5483a2be7023faa34d9f633773a1c154fbc293d5f0ae1e595b4f1d WatchSource:0}: Error finding container 03b4ac250a5483a2be7023faa34d9f633773a1c154fbc293d5f0ae1e595b4f1d: Status 404 returned error can't find the container with id 03b4ac250a5483a2be7023faa34d9f633773a1c154fbc293d5f0ae1e595b4f1d
Nov 01 06:43:53.719278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:53.719196    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc" event={"ID":"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13","Type":"ContainerStarted","Data":"9e5bb95bd412380f2adf8c8d5501721294701446eec789ba2dde8cc20967fde2"}
Nov 01 06:43:53.719278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:53.719243    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc" event={"ID":"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13","Type":"ContainerStarted","Data":"03b4ac250a5483a2be7023faa34d9f633773a1c154fbc293d5f0ae1e595b4f1d"}
Nov 01 06:43:53.831366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:53.831308    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc" podStartSLOduration=1.8312947880000001 podStartE2EDuration="1.831294788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:53.830211805 +0000 UTC m=+2045.398018656" watchObservedRunningTime="2026-11-01 06:43:53.831294788 +0000 UTC m=+2045.399101618"
Nov 01 06:43:54.726370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:54.726334    4468 generic.go:356] "Generic (PLEG): container finished" podID="e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" containerID="9e5bb95bd412380f2adf8c8d5501721294701446eec789ba2dde8cc20967fde2" exitCode=0
Nov 01 06:43:54.726785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:54.726421    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc" event={"ID":"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13","Type":"ContainerDied","Data":"9e5bb95bd412380f2adf8c8d5501721294701446eec789ba2dde8cc20967fde2"}
Nov 01 06:43:55.872675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.872648    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:55.931671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.931629    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd\" (UniqueName: \"kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd\") pod \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") "
Nov 01 06:43:55.931820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.931689    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca\") pod \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") "
Nov 01 06:43:55.931820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.931719    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca\") pod \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\" (UID: \"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13\") "
Nov 01 06:43:55.932132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.932100    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca" pod "e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" (UID: "e1f55a17-d5f1-44f4-b24b-ebb04fff4f13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:43:55.932279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.932261    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:55.933820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.933800    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca" pod "e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" (UID: "e1f55a17-d5f1-44f4-b24b-ebb04fff4f13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:43:55.933974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:55.933954    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd" pod "e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" (UID: "e1f55a17-d5f1-44f4-b24b-ebb04fff4f13"). InnerVolumeSpecName "kube-api-access-npftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:43:56.033768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:56.033698    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npftd\" (UniqueName: \"kubernetes.io/projected/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-kube-api-access-npftd\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:56.033768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:56.033725    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:56.739194 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:56.739158    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"
Nov 01 06:43:56.739194 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:56.739181    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc" event={"ID":"e1f55a17-d5f1-44f4-b24b-ebb04fff4f13","Type":"ContainerDied","Data":"03b4ac250a5483a2be7023faa34d9f633773a1c154fbc293d5f0ae1e595b4f1d"}
Nov 01 06:43:56.739418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:56.739225    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b4ac250a5483a2be7023faa34d9f633773a1c154fbc293d5f0ae1e595b4f1d"
Nov 01 06:43:57.864146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:43:57.864118    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:44:02.947345 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:44:02.947306    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:44:11.358202 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.358162    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"]
Nov 01 06:44:11.358986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.358964    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" containerName="revoke-keys"
Nov 01 06:44:11.359030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.358991    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" containerName="revoke-keys"
Nov 01 06:44:11.359190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.359177    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" containerName="revoke-keys"
Nov 01 06:44:11.362872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.362853    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.405635 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.405599    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.405799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.405654    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5959t\" (UniqueName: \"kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.405799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.405782    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.418864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.418835    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"]
Nov 01 06:44:11.464798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.464767    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:44:11.464798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.464783    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:44:11.464970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.464795    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:44:11.464970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.464848    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:44:11.506762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.506735    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.506899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.506773    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5959t\" (UniqueName: \"kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.506899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.506811    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.507354 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.507331    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.509263 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.509244    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.562827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.562800    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5959t\" (UniqueName: \"kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t\") pod \"maas-api-revoke-sub-ce1d3503-cv2cs\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:11.676024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:11.675956    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:12.046575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:12.046546    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"]
Nov 01 06:44:12.047597 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:44:12.047572    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedfecb8_266c_4835_be8f_d4a39013cd39.slice/crio-ca5fcbfe4ca8d200ab65e291dccda6a05ca983054bca6169ea452d6256640429 WatchSource:0}: Error finding container ca5fcbfe4ca8d200ab65e291dccda6a05ca983054bca6169ea452d6256640429: Status 404 returned error can't find the container with id ca5fcbfe4ca8d200ab65e291dccda6a05ca983054bca6169ea452d6256640429
Nov 01 06:44:12.836269 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:12.836226    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs" event={"ID":"bedfecb8-266c-4835-be8f-d4a39013cd39","Type":"ContainerStarted","Data":"435e800e4e2aad4c69ae2366081420d8317188cb3b45254a0d3c11a444dfa385"}
Nov 01 06:44:12.836729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:12.836291    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs" event={"ID":"bedfecb8-266c-4835-be8f-d4a39013cd39","Type":"ContainerStarted","Data":"ca5fcbfe4ca8d200ab65e291dccda6a05ca983054bca6169ea452d6256640429"}
Nov 01 06:44:12.948691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:12.948645    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs" podStartSLOduration=1.94862891 podStartE2EDuration="1.94862891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:12.948215321 +0000 UTC m=+2064.516022151" watchObservedRunningTime="2026-11-01 06:44:12.94862891 +0000 UTC m=+2064.516435741"
Nov 01 06:44:13.844074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:13.844035    4468 generic.go:356] "Generic (PLEG): container finished" podID="bedfecb8-266c-4835-be8f-d4a39013cd39" containerID="435e800e4e2aad4c69ae2366081420d8317188cb3b45254a0d3c11a444dfa385" exitCode=0
Nov 01 06:44:13.844576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:13.844093    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs" event={"ID":"bedfecb8-266c-4835-be8f-d4a39013cd39","Type":"ContainerDied","Data":"435e800e4e2aad4c69ae2366081420d8317188cb3b45254a0d3c11a444dfa385"}
Nov 01 06:44:14.985404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:14.985381    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:15.046820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.046787    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca\") pod \"bedfecb8-266c-4835-be8f-d4a39013cd39\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") "
Nov 01 06:44:15.046972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.046875    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t\" (UniqueName: \"kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t\") pod \"bedfecb8-266c-4835-be8f-d4a39013cd39\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") "
Nov 01 06:44:15.047040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.047001    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca\") pod \"bedfecb8-266c-4835-be8f-d4a39013cd39\" (UID: \"bedfecb8-266c-4835-be8f-d4a39013cd39\") "
Nov 01 06:44:15.047130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.047105    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca" pod "bedfecb8-266c-4835-be8f-d4a39013cd39" (UID: "bedfecb8-266c-4835-be8f-d4a39013cd39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:15.047317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.047300    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:15.049041 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.049020    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca" pod "bedfecb8-266c-4835-be8f-d4a39013cd39" (UID: "bedfecb8-266c-4835-be8f-d4a39013cd39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:44:15.049104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.049044    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t" pod "bedfecb8-266c-4835-be8f-d4a39013cd39" (UID: "bedfecb8-266c-4835-be8f-d4a39013cd39"). InnerVolumeSpecName "kube-api-access-5959t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:15.148068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.147995    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5959t\" (UniqueName: \"kubernetes.io/projected/bedfecb8-266c-4835-be8f-d4a39013cd39-kube-api-access-5959t\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:15.148068 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.148026    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedfecb8-266c-4835-be8f-d4a39013cd39-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:15.857306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.857280    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"
Nov 01 06:44:15.857491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.857325    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs" event={"ID":"bedfecb8-266c-4835-be8f-d4a39013cd39","Type":"ContainerDied","Data":"ca5fcbfe4ca8d200ab65e291dccda6a05ca983054bca6169ea452d6256640429"}
Nov 01 06:44:15.857491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:15.857393    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5fcbfe4ca8d200ab65e291dccda6a05ca983054bca6169ea452d6256640429"
Nov 01 06:44:15.871142 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:44:15.871118    4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedfecb8_266c_4835_be8f_d4a39013cd39.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 06:44:22.273393 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.273356    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"]
Nov 01 06:44:22.275798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.273938    4468 cpu_manager.go:414] "containerMap: removing container" podUID="bedfecb8-266c-4835-be8f-d4a39013cd39" containerName="revoke-keys"
Nov 01 06:44:22.275798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.273950    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedfecb8-266c-4835-be8f-d4a39013cd39" containerName="revoke-keys"
Nov 01 06:44:22.275798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.274044    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedfecb8-266c-4835-be8f-d4a39013cd39" containerName="revoke-keys"
Nov 01 06:44:22.277195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.277179    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.333305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.333281    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"]
Nov 01 06:44:22.372671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.372644    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.372824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.372755    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.372824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.372809    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8cj\" (UniqueName: \"kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.378938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.378911    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:44:22.379031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.378937    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:44:22.379031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.378978    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:44:22.379031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.379022    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:44:22.474313 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.474282    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.474453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.474334    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8cj\" (UniqueName: \"kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.474453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.474420    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.475067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.475043    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.476921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.476894    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.530737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.530672    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8cj\" (UniqueName: \"kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj\") pod \"maas-api-revoke-sub-bbc38c69-hgtfn\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.590606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.590573    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:22.963864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:22.963839    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"]
Nov 01 06:44:22.965385 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:44:22.965358    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeedf05_bc6e_4086_8672_98cf79d6a139.slice/crio-d92621bd1f1bcf30ba6c26d42c0ccb62f1ae2ec701681687e72fed50c687ea5f WatchSource:0}: Error finding container d92621bd1f1bcf30ba6c26d42c0ccb62f1ae2ec701681687e72fed50c687ea5f: Status 404 returned error can't find the container with id d92621bd1f1bcf30ba6c26d42c0ccb62f1ae2ec701681687e72fed50c687ea5f
Nov 01 06:44:23.935521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:23.935453    4468 generic.go:356] "Generic (PLEG): container finished" podID="ceeedf05-bc6e-4086-8672-98cf79d6a139" containerID="db246e1ec6a3971a8710e9fd472ba104f9d942398a84861ff1c2d18ed6109a49" exitCode=0
Nov 01 06:44:23.935521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:23.935503    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn" event={"ID":"ceeedf05-bc6e-4086-8672-98cf79d6a139","Type":"ContainerDied","Data":"db246e1ec6a3971a8710e9fd472ba104f9d942398a84861ff1c2d18ed6109a49"}
Nov 01 06:44:23.936286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:23.935566    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn" event={"ID":"ceeedf05-bc6e-4086-8672-98cf79d6a139","Type":"ContainerStarted","Data":"d92621bd1f1bcf30ba6c26d42c0ccb62f1ae2ec701681687e72fed50c687ea5f"}
Nov 01 06:44:25.085071 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.085049    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:25.105217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.105195    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj\" (UniqueName: \"kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj\") pod \"ceeedf05-bc6e-4086-8672-98cf79d6a139\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") "
Nov 01 06:44:25.105318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.105306    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca\") pod \"ceeedf05-bc6e-4086-8672-98cf79d6a139\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") "
Nov 01 06:44:25.105415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.105405    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca\") pod \"ceeedf05-bc6e-4086-8672-98cf79d6a139\" (UID: \"ceeedf05-bc6e-4086-8672-98cf79d6a139\") "
Nov 01 06:44:25.105772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.105743    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca" pod "ceeedf05-bc6e-4086-8672-98cf79d6a139" (UID: "ceeedf05-bc6e-4086-8672-98cf79d6a139"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:25.107291 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.107273    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj" pod "ceeedf05-bc6e-4086-8672-98cf79d6a139" (UID: "ceeedf05-bc6e-4086-8672-98cf79d6a139"). InnerVolumeSpecName "kube-api-access-js8cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:25.107385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.107336    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca" pod "ceeedf05-bc6e-4086-8672-98cf79d6a139" (UID: "ceeedf05-bc6e-4086-8672-98cf79d6a139"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:44:25.207193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.207128    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:25.207193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.207155    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8cj\" (UniqueName: \"kubernetes.io/projected/ceeedf05-bc6e-4086-8672-98cf79d6a139-kube-api-access-js8cj\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:25.207193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.207166    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceeedf05-bc6e-4086-8672-98cf79d6a139-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:25.947720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.947683    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn" event={"ID":"ceeedf05-bc6e-4086-8672-98cf79d6a139","Type":"ContainerDied","Data":"d92621bd1f1bcf30ba6c26d42c0ccb62f1ae2ec701681687e72fed50c687ea5f"}
Nov 01 06:44:25.947915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.947745    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92621bd1f1bcf30ba6c26d42c0ccb62f1ae2ec701681687e72fed50c687ea5f"
Nov 01 06:44:25.947915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:25.947691    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"
Nov 01 06:44:32.946790 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:44:32.946755    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:44:35.236651 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:35.236614    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:44:45.082049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:45.082019    4468 generic.go:356] "Generic (PLEG): container finished" podID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerID="7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673" exitCode=0
Nov 01 06:44:45.082461 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:45.082094    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" event={"ID":"49078d4c-aabb-44cf-bea4-6b9a67de3f8b","Type":"ContainerDied","Data":"7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"}
Nov 01 06:44:45.082544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:45.082457    4468 scope.go:122] "RemoveContainer" containerID="7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"
Nov 01 06:44:45.083775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:45.083748    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:44:45.083846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:45.083813    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:44:45.083882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:45.083848    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:44:46.089187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:46.089156    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" event={"ID":"49078d4c-aabb-44cf-bea4-6b9a67de3f8b","Type":"ContainerStarted","Data":"3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7"}
Nov 01 06:44:46.391525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:46.391419    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:44:46.392638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:46.392616    4468 patch_prober.go:29] interesting pod/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 06:44:46.392711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:46.392669    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 06:44:47.391759 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:47.391725    4468 patch_prober.go:29] interesting pod/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 06:44:47.392178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:47.391816    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 06:44:48.391459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:48.391426    4468 patch_prober.go:29] interesting pod/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 06:44:48.391640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:48.391505    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 06:44:48.627487 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:48.626611    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:44:49.391412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:49.391382    4468 patch_prober.go:29] interesting pod/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 06:44:49.391630 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:49.391437    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 06:44:50.391790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:50.391752    4468 patch_prober.go:29] interesting pod/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 06:44:50.392269 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:50.391818    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 06:44:51.395663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:51.395633    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:44:51.396667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:51.396650    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:44:56.374377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:56.374330    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"]
Nov 01 06:44:57.275946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.275915    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"]
Nov 01 06:44:57.276521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.276507    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ceeedf05-bc6e-4086-8672-98cf79d6a139" containerName="revoke-keys"
Nov 01 06:44:57.276571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.276523    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceeedf05-bc6e-4086-8672-98cf79d6a139" containerName="revoke-keys"
Nov 01 06:44:57.276632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.276621    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceeedf05-bc6e-4086-8672-98cf79d6a139" containerName="revoke-keys"
Nov 01 06:44:57.278960 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.278944    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330209    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330310    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330344    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330395    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330421    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330521    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6j2\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.334495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.330558    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.350490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.342056    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8a9760-openshift-default-dockercfg-ps4qr\""
Nov 01 06:44:57.350490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.343012    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.350490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.343053    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.399389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.399354    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"]
Nov 01 06:44:57.444013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.443982    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444046    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444070    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444159    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444232    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444261    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444312    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444335    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444404    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6j2\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444610 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444573    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444754    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444853 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444796    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.444960 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.444941    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.445188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.445165    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.446871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.446849    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.447385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.447361    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.500830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.500798    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.501090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.501066    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6j2\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2\") pod \"e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") " pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.590507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.590443    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:57.681971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:57.681941    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:44:58.018800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.018774    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"]
Nov 01 06:44:58.019995 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:44:58.019970    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe32ed1d_772c_4a98_926a_653cb28329c1.slice/crio-e40e03dd4c8a4de94de7439956d3fe0670c3f9a7f26b575c370345bf49ea35ba WatchSource:0}: Error finding container e40e03dd4c8a4de94de7439956d3fe0670c3f9a7f26b575c370345bf49ea35ba: Status 404 returned error can't find the container with id e40e03dd4c8a4de94de7439956d3fe0670c3f9a7f26b575c370345bf49ea35ba
Nov 01 06:44:58.022964 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.022928    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:44:58.023085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.023015    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:44:58.023085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.023045    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:44:58.192360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.192326    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" event={"ID":"be32ed1d-772c-4a98-926a-653cb28329c1","Type":"ContainerStarted","Data":"7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"}
Nov 01 06:44:58.192594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.192387    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" event={"ID":"be32ed1d-772c-4a98-926a-653cb28329c1","Type":"ContainerStarted","Data":"e40e03dd4c8a4de94de7439956d3fe0670c3f9a7f26b575c370345bf49ea35ba"}
Nov 01 06:44:58.319489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.313123    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podStartSLOduration=1.313104786 podStartE2EDuration="1.313104786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:58.312602931 +0000 UTC m=+2109.880409787" watchObservedRunningTime="2026-11-01 06:44:58.313104786 +0000 UTC m=+2109.880911616"
Nov 01 06:44:58.597362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:58.597291    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:44:59.597402 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:44:59.597322    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:45:00.167126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.167091    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"]
Nov 01 06:45:00.170602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.170575    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"]
Nov 01 06:45:00.170752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.170695    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.176024 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.176004    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"]
Nov 01 06:45:00.176141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.176109    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.184439 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.184410    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.184582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.184563    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhf9\" (UniqueName: \"kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.184627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.184598    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2c7\" (UniqueName: \"kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.184681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.184650    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.189822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.189799    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"]
Nov 01 06:45:00.189947 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.189930    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.192870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.192847    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.193201 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.193187    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"]
Nov 01 06:45:00.196321 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.196299    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"]
Nov 01 06:45:00.196548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.196528    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.199451 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.199434    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.208587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.208555    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:45:00.209606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.209586    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:45:00.215945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.215924    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"]
Nov 01 06:45:00.219535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.219518    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.224623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.224579    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:45:00.224772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.224753    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:45:00.224986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.224928    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:45:00.228613 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.228584    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"]
Nov 01 06:45:00.266850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.266818    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"]
Nov 01 06:45:00.274152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.273368    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"]
Nov 01 06:45:00.280176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.280146    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"]
Nov 01 06:45:00.286537 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.285183    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"]
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.286930    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.286981    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt272\" (UniqueName: \"kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287025    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5q4l\" (UniqueName: \"kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287173    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdt5j\" (UniqueName: \"kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287224    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287252    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287282    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287405    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhf9\" (UniqueName: \"kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287431    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287480    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2c7\" (UniqueName: \"kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287514    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bh6\" (UniqueName: \"kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287563    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqc65\" (UniqueName: \"kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287612    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.288493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.287664    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.289326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.288702    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.290201 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.290176    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.292022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.292002    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"]
Nov 01 06:45:00.294664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.294641    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"]
Nov 01 06:45:00.345189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.345153    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2c7\" (UniqueName: \"kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.345370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.345156    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhf9\" (UniqueName: \"kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.388647 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388612    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.388828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388656    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bh6\" (UniqueName: \"kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.388828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388692    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqc65\" (UniqueName: \"kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.388828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388739    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.388828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388769    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.389056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388883    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt272\" (UniqueName: \"kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.389056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.388940    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5q4l\" (UniqueName: \"kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.389180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389058    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdt5j\" (UniqueName: \"kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.389180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389092    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.389180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389118    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.389510 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389482    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.389627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389535    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.389627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389556    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.389780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389764    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.389833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.389817    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.445669 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.445593    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdt5j\" (UniqueName: \"kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.445948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.445924    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqc65\" (UniqueName: \"kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.446077 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.445994    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bh6\" (UniqueName: \"kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.446311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.446293    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt272\" (UniqueName: \"kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.446524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.446506    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5q4l\" (UniqueName: \"kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.487155 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.487123    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:00.495244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.495224    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:00.510521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.509657    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:00.515134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.515111    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:00.523516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.523490    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:00.534145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.534047    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:00.548461 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.548398    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:00.879414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.879380    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"]
Nov 01 06:45:00.880506 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:00.880439    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bcc87c_b6d6_4137_95bd_9b7f13f36494.slice/crio-68662f545915b8db9aeb9175c4a4e41442151069e9214817ab295e7c1f7cf103 WatchSource:0}: Error finding container 68662f545915b8db9aeb9175c4a4e41442151069e9214817ab295e7c1f7cf103: Status 404 returned error can't find the container with id 68662f545915b8db9aeb9175c4a4e41442151069e9214817ab295e7c1f7cf103
Nov 01 06:45:00.921001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.920977    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"]
Nov 01 06:45:00.922183 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:00.922154    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e56c70_2174_4f4c_acc6_1756dcd6fc50.slice/crio-a023f9853c5ed176dbea9df017621c60da426b748d9e2ad3cd7028b77f8b8910 WatchSource:0}: Error finding container a023f9853c5ed176dbea9df017621c60da426b748d9e2ad3cd7028b77f8b8910: Status 404 returned error can't find the container with id a023f9853c5ed176dbea9df017621c60da426b748d9e2ad3cd7028b77f8b8910
Nov 01 06:45:00.930960 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.930937    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"]
Nov 01 06:45:00.935275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:00.935247    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"]
Nov 01 06:45:00.935379 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:00.935361    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9afc4bb_a453_4f23_ac87_7a237dfae755.slice/crio-3c880473c500c241839f1ab05e35f318f8f5d588091797eec314b9bcb492ec1f WatchSource:0}: Error finding container 3c880473c500c241839f1ab05e35f318f8f5d588091797eec314b9bcb492ec1f: Status 404 returned error can't find the container with id 3c880473c500c241839f1ab05e35f318f8f5d588091797eec314b9bcb492ec1f
Nov 01 06:45:00.936636 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:00.936615    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5937e71_4463_4f86_8b5f_9f78093f57d7.slice/crio-d42be314d1428119d1139371842e47e5f706a9692a8d24148203e0fbdeedee69 WatchSource:0}: Error finding container d42be314d1428119d1139371842e47e5f706a9692a8d24148203e0fbdeedee69: Status 404 returned error can't find the container with id d42be314d1428119d1139371842e47e5f706a9692a8d24148203e0fbdeedee69
Nov 01 06:45:01.004984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.004949    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"]
Nov 01 06:45:01.009374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.009347    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"]
Nov 01 06:45:01.106805 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.106777    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"]
Nov 01 06:45:01.107793 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:01.107769    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e56b16_833d_4614_b9b2_f11eb5ea7b4e.slice/crio-cee531811b3263519dec79bb131db2330f6dbb60e215c5bec230122181a7c3a2 WatchSource:0}: Error finding container cee531811b3263519dec79bb131db2330f6dbb60e215c5bec230122181a7c3a2: Status 404 returned error can't find the container with id cee531811b3263519dec79bb131db2330f6dbb60e215c5bec230122181a7c3a2
Nov 01 06:45:01.116603 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:01.116571    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6613df73_2ad6_4ef7_9623_b4499983d52e.slice/crio-741564678dc0903574b3def86f1d6e0ec2d6f2fa77671db522f16f56bf85a0c9 WatchSource:0}: Error finding container 741564678dc0903574b3def86f1d6e0ec2d6f2fa77671db522f16f56bf85a0c9: Status 404 returned error can't find the container with id 741564678dc0903574b3def86f1d6e0ec2d6f2fa77671db522f16f56bf85a0c9
Nov 01 06:45:01.216316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.216283    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerStarted","Data":"3c880473c500c241839f1ab05e35f318f8f5d588091797eec314b9bcb492ec1f"}
Nov 01 06:45:01.217886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.217862    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerStarted","Data":"68662f545915b8db9aeb9175c4a4e41442151069e9214817ab295e7c1f7cf103"}
Nov 01 06:45:01.219189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.219168    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerStarted","Data":"a023f9853c5ed176dbea9df017621c60da426b748d9e2ad3cd7028b77f8b8910"}
Nov 01 06:45:01.220507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.220450    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerStarted","Data":"d42be314d1428119d1139371842e47e5f706a9692a8d24148203e0fbdeedee69"}
Nov 01 06:45:01.221701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.221681    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" event={"ID":"51e02628-a7d7-42c6-bbe9-84bb96c18c51","Type":"ContainerStarted","Data":"256b25ebd3588f9b1bd5a1b565f6db9f0413e23eb0c8d2739e3be0d805836603"}
Nov 01 06:45:01.223008 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.222988    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" event={"ID":"6613df73-2ad6-4ef7-9623-b4499983d52e","Type":"ContainerStarted","Data":"741564678dc0903574b3def86f1d6e0ec2d6f2fa77671db522f16f56bf85a0c9"}
Nov 01 06:45:01.224084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:01.224066    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerStarted","Data":"cee531811b3263519dec79bb131db2330f6dbb60e215c5bec230122181a7c3a2"}
Nov 01 06:45:02.233303 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.233269    4468 generic.go:356] "Generic (PLEG): container finished" podID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerID="a73265813adc0345fccfdaa90bef7ff2a713e8b2bc8132a644cd708c8c07e0c4" exitCode=60
Nov 01 06:45:02.233820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.233361    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerDied","Data":"a73265813adc0345fccfdaa90bef7ff2a713e8b2bc8132a644cd708c8c07e0c4"}
Nov 01 06:45:02.233820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.233645    4468 scope.go:122] "RemoveContainer" containerID="a73265813adc0345fccfdaa90bef7ff2a713e8b2bc8132a644cd708c8c07e0c4"
Nov 01 06:45:02.235072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.235054    4468 generic.go:356] "Generic (PLEG): container finished" podID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerID="c7f38377bb64e46a5715c271d980deefd96cbab88b6cd34c96e1075fc44ba541" exitCode=60
Nov 01 06:45:02.235189 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.235133    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerDied","Data":"c7f38377bb64e46a5715c271d980deefd96cbab88b6cd34c96e1075fc44ba541"}
Nov 01 06:45:02.235340 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.235323    4468 scope.go:122] "RemoveContainer" containerID="c7f38377bb64e46a5715c271d980deefd96cbab88b6cd34c96e1075fc44ba541"
Nov 01 06:45:02.236915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.236857    4468 generic.go:356] "Generic (PLEG): container finished" podID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerID="2012069795af9d3452654d9450ebc837ecf577085cf7fb1cbe16dbddbcc1919f" exitCode=60
Nov 01 06:45:02.236971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.236938    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" event={"ID":"51e02628-a7d7-42c6-bbe9-84bb96c18c51","Type":"ContainerDied","Data":"2012069795af9d3452654d9450ebc837ecf577085cf7fb1cbe16dbddbcc1919f"}
Nov 01 06:45:02.237161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.237142    4468 scope.go:122] "RemoveContainer" containerID="2012069795af9d3452654d9450ebc837ecf577085cf7fb1cbe16dbddbcc1919f"
Nov 01 06:45:02.238620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.238599    4468 generic.go:356] "Generic (PLEG): container finished" podID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerID="5d96288dc537a7178b16eae2a882f23bc6b913c118c3e04c35d64c73f8218a44" exitCode=60
Nov 01 06:45:02.238699 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.238652    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" event={"ID":"6613df73-2ad6-4ef7-9623-b4499983d52e","Type":"ContainerDied","Data":"5d96288dc537a7178b16eae2a882f23bc6b913c118c3e04c35d64c73f8218a44"}
Nov 01 06:45:02.238845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.238829    4468 scope.go:122] "RemoveContainer" containerID="5d96288dc537a7178b16eae2a882f23bc6b913c118c3e04c35d64c73f8218a44"
Nov 01 06:45:02.240332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.240316    4468 generic.go:356] "Generic (PLEG): container finished" podID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerID="4aeffa42f12d29be8466f29be8876cdb1039114fb3faf23431ffe1cbf12e24f5" exitCode=60
Nov 01 06:45:02.240417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.240389    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerDied","Data":"4aeffa42f12d29be8466f29be8876cdb1039114fb3faf23431ffe1cbf12e24f5"}
Nov 01 06:45:02.240649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.240636    4468 scope.go:122] "RemoveContainer" containerID="4aeffa42f12d29be8466f29be8876cdb1039114fb3faf23431ffe1cbf12e24f5"
Nov 01 06:45:02.242483 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.242443    4468 generic.go:356] "Generic (PLEG): container finished" podID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerID="9827ced132c0004c8e03b2853e41043428ce556514e880d4f3a03f5cfee3376a" exitCode=60
Nov 01 06:45:02.242590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.242525    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerDied","Data":"9827ced132c0004c8e03b2853e41043428ce556514e880d4f3a03f5cfee3376a"}
Nov 01 06:45:02.242742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.242727    4468 scope.go:122] "RemoveContainer" containerID="9827ced132c0004c8e03b2853e41043428ce556514e880d4f3a03f5cfee3376a"
Nov 01 06:45:02.244665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.244646    4468 generic.go:356] "Generic (PLEG): container finished" podID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerID="5a444499e1a1940a60636234b20ce7f1c6381329cc2a1accced91fbae37b4cf2" exitCode=60
Nov 01 06:45:02.244768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.244732    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerDied","Data":"5a444499e1a1940a60636234b20ce7f1c6381329cc2a1accced91fbae37b4cf2"}
Nov 01 06:45:02.244854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:02.244841    4468 scope.go:122] "RemoveContainer" containerID="5a444499e1a1940a60636234b20ce7f1c6381329cc2a1accced91fbae37b4cf2"
Nov 01 06:45:02.947791 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:02.947750    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:45:03.253742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.253703    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerStarted","Data":"7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444"}
Nov 01 06:45:03.255623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.255588    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerStarted","Data":"6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45"}
Nov 01 06:45:03.257361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.257338    4468 generic.go:356] "Generic (PLEG): container finished" podID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerID="71f96900675dab63e080440dd91f88340f5c92989de17a8c09cd670c21d1b70a" exitCode=60
Nov 01 06:45:03.257458 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.257434    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" event={"ID":"51e02628-a7d7-42c6-bbe9-84bb96c18c51","Type":"ContainerDied","Data":"71f96900675dab63e080440dd91f88340f5c92989de17a8c09cd670c21d1b70a"}
Nov 01 06:45:03.257521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.257491    4468 scope.go:122] "RemoveContainer" containerID="2012069795af9d3452654d9450ebc837ecf577085cf7fb1cbe16dbddbcc1919f"
Nov 01 06:45:03.257626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.257610    4468 scope.go:122] "RemoveContainer" containerID="71f96900675dab63e080440dd91f88340f5c92989de17a8c09cd670c21d1b70a"
Nov 01 06:45:03.257905 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:03.257848    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt_odh-ai-gateway-infra(51e02628-a7d7-42c6-bbe9-84bb96c18c51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51"
Nov 01 06:45:03.259433 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.259410    4468 generic.go:356] "Generic (PLEG): container finished" podID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerID="33f18a2b7bcbbd2635685317989c398de8e01c84adaed04b09b0e20ff87176ef" exitCode=60
Nov 01 06:45:03.259563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.259495    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" event={"ID":"6613df73-2ad6-4ef7-9623-b4499983d52e","Type":"ContainerDied","Data":"33f18a2b7bcbbd2635685317989c398de8e01c84adaed04b09b0e20ff87176ef"}
Nov 01 06:45:03.259729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.259714    4468 scope.go:122] "RemoveContainer" containerID="33f18a2b7bcbbd2635685317989c398de8e01c84adaed04b09b0e20ff87176ef"
Nov 01 06:45:03.259986 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:03.259964    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb_odh-ai-gateway-infra(6613df73-2ad6-4ef7-9623-b4499983d52e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e"
Nov 01 06:45:03.261316 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.261297    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerStarted","Data":"1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed"}
Nov 01 06:45:03.263178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.263159    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerStarted","Data":"a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a"}
Nov 01 06:45:03.264896 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.264879    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerStarted","Data":"61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b"}
Nov 01 06:45:03.283384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.283354    4468 scope.go:122] "RemoveContainer" containerID="5d96288dc537a7178b16eae2a882f23bc6b913c118c3e04c35d64c73f8218a44"
Nov 01 06:45:03.367654 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.367606    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" podStartSLOduration=3.367592316 podStartE2EDuration="3.367592316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.364749725 +0000 UTC m=+2114.932556553" watchObservedRunningTime="2026-11-01 06:45:03.367592316 +0000 UTC m=+2114.935399145"
Nov 01 06:45:03.477412 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.477307    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" podStartSLOduration=3.477291177 podStartE2EDuration="3.477291177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.475568537 +0000 UTC m=+2115.043375365" watchObservedRunningTime="2026-11-01 06:45:03.477291177 +0000 UTC m=+2115.045098050"
Nov 01 06:45:03.591670 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.591621    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" podStartSLOduration=3.591606679 podStartE2EDuration="3.591606679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.589212148 +0000 UTC m=+2115.157018979" watchObservedRunningTime="2026-11-01 06:45:03.591606679 +0000 UTC m=+2115.159413508"
Nov 01 06:45:03.704205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.704145    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" podStartSLOduration=3.704129206 podStartE2EDuration="3.704129206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.701173985 +0000 UTC m=+2115.268980826" watchObservedRunningTime="2026-11-01 06:45:03.704129206 +0000 UTC m=+2115.271936034"
Nov 01 06:45:03.815589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:03.815533    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" podStartSLOduration=3.815517814 podStartE2EDuration="3.815517814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.812710759 +0000 UTC m=+2115.380517591" watchObservedRunningTime="2026-11-01 06:45:03.815517814 +0000 UTC m=+2115.383324833"
Nov 01 06:45:04.275265 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.275177    4468 scope.go:122] "RemoveContainer" containerID="33f18a2b7bcbbd2635685317989c398de8e01c84adaed04b09b0e20ff87176ef"
Nov 01 06:45:04.275762 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.275613    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb_odh-ai-gateway-infra(6613df73-2ad6-4ef7-9623-b4499983d52e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e"
Nov 01 06:45:04.276800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.276773    4468 generic.go:356] "Generic (PLEG): container finished" podID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerID="1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed" exitCode=60
Nov 01 06:45:04.276935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.276855    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerDied","Data":"1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed"}
Nov 01 06:45:04.276935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.276919    4468 scope.go:122] "RemoveContainer" containerID="4aeffa42f12d29be8466f29be8876cdb1039114fb3faf23431ffe1cbf12e24f5"
Nov 01 06:45:04.277114 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.277098    4468 scope.go:122] "RemoveContainer" containerID="1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed"
Nov 01 06:45:04.277409 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.277386    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t_odh-ai-gateway-infra(98e56b16-833d-4614-b9b2-f11eb5ea7b4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e"
Nov 01 06:45:04.278907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.278890    4468 generic.go:356] "Generic (PLEG): container finished" podID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerID="a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a" exitCode=60
Nov 01 06:45:04.278986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.278952    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerDied","Data":"a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a"}
Nov 01 06:45:04.279210 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.279197    4468 scope.go:122] "RemoveContainer" containerID="a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a"
Nov 01 06:45:04.279494 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.279457    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb_odh-ai-gateway-infra(f9afc4bb-a453-4f23-ac87-7a237dfae755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755"
Nov 01 06:45:04.280783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.280764    4468 generic.go:356] "Generic (PLEG): container finished" podID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerID="61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b" exitCode=60
Nov 01 06:45:04.281094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.280879    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerDied","Data":"61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b"}
Nov 01 06:45:04.281094 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.281032    4468 scope.go:122] "RemoveContainer" containerID="61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b"
Nov 01 06:45:04.281276 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.281247    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx_odh-ai-gateway-infra(a5bcc87c-b6d6-4137-95bd-9b7f13f36494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494"
Nov 01 06:45:04.282808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.282785    4468 generic.go:356] "Generic (PLEG): container finished" podID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerID="7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444" exitCode=60
Nov 01 06:45:04.282906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.282867    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerDied","Data":"7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444"}
Nov 01 06:45:04.283090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.283073    4468 scope.go:122] "RemoveContainer" containerID="7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444"
Nov 01 06:45:04.283360 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.283328    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k_odh-ai-gateway-infra(d3e56c70-2174-4f4c-acc6-1756dcd6fc50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50"
Nov 01 06:45:04.284758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.284736    4468 generic.go:356] "Generic (PLEG): container finished" podID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerID="6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45" exitCode=60
Nov 01 06:45:04.284848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.284816    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerDied","Data":"6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45"}
Nov 01 06:45:04.286459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.286434    4468 scope.go:122] "RemoveContainer" containerID="71f96900675dab63e080440dd91f88340f5c92989de17a8c09cd670c21d1b70a"
Nov 01 06:45:04.286736 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.286712    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt_odh-ai-gateway-infra(51e02628-a7d7-42c6-bbe9-84bb96c18c51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51"
Nov 01 06:45:04.288726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.288703    4468 scope.go:122] "RemoveContainer" containerID="6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45"
Nov 01 06:45:04.288992 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:04.288971    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n_odh-ai-gateway-infra(b5937e71-4463-4f86-8b5f-9f78093f57d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7"
Nov 01 06:45:04.289730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.289716    4468 scope.go:122] "RemoveContainer" containerID="9827ced132c0004c8e03b2853e41043428ce556514e880d4f3a03f5cfee3376a"
Nov 01 06:45:04.303870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.303842    4468 scope.go:122] "RemoveContainer" containerID="5a444499e1a1940a60636234b20ce7f1c6381329cc2a1accced91fbae37b4cf2"
Nov 01 06:45:04.315830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.315809    4468 scope.go:122] "RemoveContainer" containerID="a73265813adc0345fccfdaa90bef7ff2a713e8b2bc8132a644cd708c8c07e0c4"
Nov 01 06:45:04.327818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:04.327800    4468 scope.go:122] "RemoveContainer" containerID="c7f38377bb64e46a5715c271d980deefd96cbab88b6cd34c96e1075fc44ba541"
Nov 01 06:45:05.295028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:05.295000    4468 scope.go:122] "RemoveContainer" containerID="1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed"
Nov 01 06:45:05.295540 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:05.295357    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t_odh-ai-gateway-infra(98e56b16-833d-4614-b9b2-f11eb5ea7b4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e"
Nov 01 06:45:05.296623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:05.296599    4468 scope.go:122] "RemoveContainer" containerID="a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a"
Nov 01 06:45:05.296869 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:05.296845    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb_odh-ai-gateway-infra(f9afc4bb-a453-4f23-ac87-7a237dfae755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755"
Nov 01 06:45:05.298145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:05.298127    4468 scope.go:122] "RemoveContainer" containerID="61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b"
Nov 01 06:45:05.298382 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:05.298363    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx_odh-ai-gateway-infra(a5bcc87c-b6d6-4137-95bd-9b7f13f36494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494"
Nov 01 06:45:05.302703 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:05.300182    4468 scope.go:122] "RemoveContainer" containerID="7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444"
Nov 01 06:45:05.302703 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:05.300645    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k_odh-ai-gateway-infra(d3e56c70-2174-4f4c-acc6-1756dcd6fc50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50"
Nov 01 06:45:05.309848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:05.309827    4468 scope.go:122] "RemoveContainer" containerID="6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45"
Nov 01 06:45:05.310056 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:05.310034    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n_odh-ai-gateway-infra(b5937e71-4463-4f86-8b5f-9f78093f57d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7"
Nov 01 06:45:11.394740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.394685    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" podUID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" containerName="main" containerID="cri-o://522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e" gracePeriod=45
Nov 01 06:45:11.662692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.662634    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:45:11.757709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.757680    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache\") pod \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") "
Nov 01 06:45:11.757857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.757839    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs\" (UniqueName: \"kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs\") pod \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") "
Nov 01 06:45:11.757901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.757872    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm\") pod \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") "
Nov 01 06:45:11.757946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.757910    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg\" (UniqueName: \"kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg\") pod \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") "
Nov 01 06:45:11.757978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.757940    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache" pod "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" (UID: "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:11.757978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.757964    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir\") pod \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") "
Nov 01 06:45:11.758082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.758066    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home\") pod \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\" (UID: \"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47\") "
Nov 01 06:45:11.758294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.758264    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir" pod "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" (UID: "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:11.758294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.758283    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home" pod "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" (UID: "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:11.758825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.758805    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:11.758906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.758831    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:11.758906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.758847    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:11.760206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.760182    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm" pod "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" (UID: "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:11.760206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.760187    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs" pod "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" (UID: "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:45:11.760329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.760268    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg" pod "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" (UID: "59cdcf27-c4f1-4679-a8ec-d2e9ea73da47"). InnerVolumeSpecName "kube-api-access-c7xrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:11.859515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.859444    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:11.859515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.859512    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:11.859515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:11.859523    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xrg\" (UniqueName: \"kubernetes.io/projected/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47-kube-api-access-c7xrg\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:12.355040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.355003    4468 generic.go:356] "Generic (PLEG): container finished" podID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" containerID="522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e" exitCode=0
Nov 01 06:45:12.355214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.355043    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"
Nov 01 06:45:12.355214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.355084    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" event={"ID":"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47","Type":"ContainerDied","Data":"522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e"}
Nov 01 06:45:12.355214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.355125    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" event={"ID":"59cdcf27-c4f1-4679-a8ec-d2e9ea73da47","Type":"ContainerDied","Data":"b3658d1149c51b946229f08435ad38cd9c2835480fd3e2575a31cbf4bc83e5b4"}
Nov 01 06:45:12.355214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.355141    4468 scope.go:122] "RemoveContainer" containerID="522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e"
Nov 01 06:45:12.365572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.365553    4468 scope.go:122] "RemoveContainer" containerID="522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e"
Nov 01 06:45:12.365809 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:12.365791    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e\": container with ID starting with 522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e not found: ID does not exist" containerID="522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e"
Nov 01 06:45:12.365876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.365823    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e"} err="failed to get container status \"522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e\": rpc error: code = NotFound desc = could not find container \"522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e\": container with ID starting with 522cda8bd406a8b20c70072e9856e2c7fffa8145ab4a45d0841e463f4b1ab16e not found: ID does not exist"
Nov 01 06:45:12.517606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.517579    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"]
Nov 01 06:45:12.567132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.567103    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p"]
Nov 01 06:45:12.590359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.590328    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" path="/var/lib/kubelet/pods/59cdcf27-c4f1-4679-a8ec-d2e9ea73da47/volumes"
Nov 01 06:45:12.619000 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:12.618944    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-51aa8f/auth-test-model-51aa8f-kserve-67f6f4b485-tn46p" podUID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:45:15.583251 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:15.583221    4468 scope.go:122] "RemoveContainer" containerID="1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed"
Nov 01 06:45:15.583721 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:15.583282    4468 scope.go:122] "RemoveContainer" containerID="6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45"
Nov 01 06:45:16.382772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:16.382738    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerStarted","Data":"0d6a6f21b7a8ea175e5e4d5b9bcc89f9201a0868c1d3dad46c606fa16c07ebfb"}
Nov 01 06:45:16.384744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:16.384721    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerStarted","Data":"58f3ebaf63192d3f2b845565443980f5acb9cba2dd8c73b602832c082b3d48ff"}
Nov 01 06:45:16.583200 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:16.583171    4468 scope.go:122] "RemoveContainer" containerID="a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a"
Nov 01 06:45:16.583322 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:16.583263    4468 scope.go:122] "RemoveContainer" containerID="61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b"
Nov 01 06:45:17.398995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.398961    4468 generic.go:356] "Generic (PLEG): container finished" podID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerID="58f3ebaf63192d3f2b845565443980f5acb9cba2dd8c73b602832c082b3d48ff" exitCode=60
Nov 01 06:45:17.399203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.399028    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerDied","Data":"58f3ebaf63192d3f2b845565443980f5acb9cba2dd8c73b602832c082b3d48ff"}
Nov 01 06:45:17.399203 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.399091    4468 scope.go:122] "RemoveContainer" containerID="6a91066906688d44b655e3953f0ed266f8f778b5ffa2765966fe83b3cf7acc45"
Nov 01 06:45:17.399577 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.399550    4468 scope.go:122] "RemoveContainer" containerID="58f3ebaf63192d3f2b845565443980f5acb9cba2dd8c73b602832c082b3d48ff"
Nov 01 06:45:17.399912 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:17.399883    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n_odh-ai-gateway-infra(b5937e71-4463-4f86-8b5f-9f78093f57d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7"
Nov 01 06:45:17.401328 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.401308    4468 generic.go:356] "Generic (PLEG): container finished" podID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerID="0d6a6f21b7a8ea175e5e4d5b9bcc89f9201a0868c1d3dad46c606fa16c07ebfb" exitCode=60
Nov 01 06:45:17.401425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.401344    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerDied","Data":"0d6a6f21b7a8ea175e5e4d5b9bcc89f9201a0868c1d3dad46c606fa16c07ebfb"}
Nov 01 06:45:17.401767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.401750    4468 scope.go:122] "RemoveContainer" containerID="0d6a6f21b7a8ea175e5e4d5b9bcc89f9201a0868c1d3dad46c606fa16c07ebfb"
Nov 01 06:45:17.402115 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:17.402091    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t_odh-ai-gateway-infra(98e56b16-833d-4614-b9b2-f11eb5ea7b4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e"
Nov 01 06:45:17.403426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.403398    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerStarted","Data":"9128ed33cde11eace0deffd7e3c1f0bd700fc35cb62a09f046eca7bb34e17454"}
Nov 01 06:45:17.406734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.406712    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerStarted","Data":"4a5ac1050e98ffbe5ff0015dbbf6a887fbc0b5de0a33402afe077721954a07a0"}
Nov 01 06:45:17.416865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.416846    4468 scope.go:122] "RemoveContainer" containerID="1a9f758a7420fe23c365a7c10652267fad0af171cd226ec0fda27e56ce0ee4ed"
Nov 01 06:45:17.505536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.505511    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"]
Nov 01 06:45:17.583246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.583215    4468 scope.go:122] "RemoveContainer" containerID="7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444"
Nov 01 06:45:17.583428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.583286    4468 scope.go:122] "RemoveContainer" containerID="33f18a2b7bcbbd2635685317989c398de8e01c84adaed04b09b0e20ff87176ef"
Nov 01 06:45:17.583428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.583406    4468 scope.go:122] "RemoveContainer" containerID="71f96900675dab63e080440dd91f88340f5c92989de17a8c09cd670c21d1b70a"
Nov 01 06:45:17.618130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:17.618102    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"]
Nov 01 06:45:18.471532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.470624    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" event={"ID":"51e02628-a7d7-42c6-bbe9-84bb96c18c51","Type":"ContainerStarted","Data":"3796766161a12fe2e2179d0b7639c4ad591dd42fdd0f574001e9f9707a045a1d"}
Nov 01 06:45:18.474544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.474503    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" event={"ID":"6613df73-2ad6-4ef7-9623-b4499983d52e","Type":"ContainerStarted","Data":"b5cdb5e162d1c54267c4791dc1ffc507580dc703da00db220840f80a8dc85baa"}
Nov 01 06:45:18.481113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.481070    4468 generic.go:356] "Generic (PLEG): container finished" podID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerID="9128ed33cde11eace0deffd7e3c1f0bd700fc35cb62a09f046eca7bb34e17454" exitCode=60
Nov 01 06:45:18.481278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.481164    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerDied","Data":"9128ed33cde11eace0deffd7e3c1f0bd700fc35cb62a09f046eca7bb34e17454"}
Nov 01 06:45:18.481278 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.481216    4468 scope.go:122] "RemoveContainer" containerID="a884ff804ca39e3924e195c89cda9a662c74b2df53dc8c6be6487d2ee4443b0a"
Nov 01 06:45:18.481735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.481714    4468 scope.go:122] "RemoveContainer" containerID="9128ed33cde11eace0deffd7e3c1f0bd700fc35cb62a09f046eca7bb34e17454"
Nov 01 06:45:18.482102 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:18.482074    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb_odh-ai-gateway-infra(f9afc4bb-a453-4f23-ac87-7a237dfae755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755"
Nov 01 06:45:18.498696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.491414    4468 generic.go:356] "Generic (PLEG): container finished" podID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerID="4a5ac1050e98ffbe5ff0015dbbf6a887fbc0b5de0a33402afe077721954a07a0" exitCode=60
Nov 01 06:45:18.498696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.491698    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerDied","Data":"4a5ac1050e98ffbe5ff0015dbbf6a887fbc0b5de0a33402afe077721954a07a0"}
Nov 01 06:45:18.498696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.492136    4468 scope.go:122] "RemoveContainer" containerID="4a5ac1050e98ffbe5ff0015dbbf6a887fbc0b5de0a33402afe077721954a07a0"
Nov 01 06:45:18.498696 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:18.492525    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx_odh-ai-gateway-infra(a5bcc87c-b6d6-4137-95bd-9b7f13f36494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494"
Nov 01 06:45:18.498696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.497090    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerStarted","Data":"d1ff3ccaa9183f4477efc9d1daa52b265303f74a86e1382448f0f0da7a41ebe7"}
Nov 01 06:45:18.513318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.513290    4468 scope.go:122] "RemoveContainer" containerID="61cc772a6fb75c9aaf27957e9dde672a598a318fedbc217b2252956e34e9c81b"
Nov 01 06:45:18.704172 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.704144    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:18.712529 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.712418    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" podStartSLOduration=18.712404406 podStartE2EDuration="18.712404406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:18.590717305 +0000 UTC m=+2130.158524134" watchObservedRunningTime="2026-11-01 06:45:18.712404406 +0000 UTC m=+2130.280211244"
Nov 01 06:45:18.785033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.785007    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"]
Nov 01 06:45:18.848053 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.848027    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:18.851330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.851308    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca\") pod \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") "
Nov 01 06:45:18.851494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.851457    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65\" (UniqueName: \"kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65\") pod \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\" (UID: \"98e56b16-833d-4614-b9b2-f11eb5ea7b4e\") "
Nov 01 06:45:18.851712 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.851692    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca" pod "98e56b16-833d-4614-b9b2-f11eb5ea7b4e" (UID: "98e56b16-833d-4614-b9b2-f11eb5ea7b4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:18.851969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.851955    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:18.853550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.853530    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65" pod "98e56b16-833d-4614-b9b2-f11eb5ea7b4e" (UID: "98e56b16-833d-4614-b9b2-f11eb5ea7b4e"). InnerVolumeSpecName "kube-api-access-bqc65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:18.896598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.896574    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"]
Nov 01 06:45:18.952381 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.952345    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j\" (UniqueName: \"kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j\") pod \"b5937e71-4463-4f86-8b5f-9f78093f57d7\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") "
Nov 01 06:45:18.952539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.952486    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca\") pod \"b5937e71-4463-4f86-8b5f-9f78093f57d7\" (UID: \"b5937e71-4463-4f86-8b5f-9f78093f57d7\") "
Nov 01 06:45:18.952865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.952843    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca" pod "b5937e71-4463-4f86-8b5f-9f78093f57d7" (UID: "b5937e71-4463-4f86-8b5f-9f78093f57d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:18.953012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.952978    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5937e71-4463-4f86-8b5f-9f78093f57d7-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:18.953012 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.953001    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqc65\" (UniqueName: \"kubernetes.io/projected/98e56b16-833d-4614-b9b2-f11eb5ea7b4e-kube-api-access-bqc65\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:18.954540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:18.954519    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j" pod "b5937e71-4463-4f86-8b5f-9f78093f57d7" (UID: "b5937e71-4463-4f86-8b5f-9f78093f57d7"). InnerVolumeSpecName "kube-api-access-cdt5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:19.043253 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.043175    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" podStartSLOduration=19.043162872 podStartE2EDuration="19.043162872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:19.040347284 +0000 UTC m=+2130.608154126" watchObservedRunningTime="2026-11-01 06:45:19.043162872 +0000 UTC m=+2130.610969767"
Nov 01 06:45:19.053961 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.053937    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdt5j\" (UniqueName: \"kubernetes.io/projected/b5937e71-4463-4f86-8b5f-9f78093f57d7-kube-api-access-cdt5j\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:19.508032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.508007    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"
Nov 01 06:45:19.508032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.508015    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t" event={"ID":"98e56b16-833d-4614-b9b2-f11eb5ea7b4e","Type":"ContainerDied","Data":"cee531811b3263519dec79bb131db2330f6dbb60e215c5bec230122181a7c3a2"}
Nov 01 06:45:19.508285 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.508076    4468 scope.go:122] "RemoveContainer" containerID="0d6a6f21b7a8ea175e5e4d5b9bcc89f9201a0868c1d3dad46c606fa16c07ebfb"
Nov 01 06:45:19.516280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.516236    4468 generic.go:356] "Generic (PLEG): container finished" podID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerID="d1ff3ccaa9183f4477efc9d1daa52b265303f74a86e1382448f0f0da7a41ebe7" exitCode=60
Nov 01 06:45:19.516842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.516430    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerDied","Data":"d1ff3ccaa9183f4477efc9d1daa52b265303f74a86e1382448f0f0da7a41ebe7"}
Nov 01 06:45:19.517368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.517350    4468 scope.go:122] "RemoveContainer" containerID="d1ff3ccaa9183f4477efc9d1daa52b265303f74a86e1382448f0f0da7a41ebe7"
Nov 01 06:45:19.518046 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:19.518010    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k_odh-ai-gateway-infra(d3e56c70-2174-4f4c-acc6-1756dcd6fc50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50"
Nov 01 06:45:19.519495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.519443    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"
Nov 01 06:45:19.519716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.519696    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n" event={"ID":"b5937e71-4463-4f86-8b5f-9f78093f57d7","Type":"ContainerDied","Data":"d42be314d1428119d1139371842e47e5f706a9692a8d24148203e0fbdeedee69"}
Nov 01 06:45:19.521418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.521373    4468 scope.go:122] "RemoveContainer" containerID="7506d8efcaeaf6bcc59d53dc69235929f3b2ed98f917ab253dffb3363d395444"
Nov 01 06:45:19.522885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.522859    4468 generic.go:356] "Generic (PLEG): container finished" podID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerID="3796766161a12fe2e2179d0b7639c4ad591dd42fdd0f574001e9f9707a045a1d" exitCode=60
Nov 01 06:45:19.522991 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.522974    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" event={"ID":"51e02628-a7d7-42c6-bbe9-84bb96c18c51","Type":"ContainerDied","Data":"3796766161a12fe2e2179d0b7639c4ad591dd42fdd0f574001e9f9707a045a1d"}
Nov 01 06:45:19.523375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.523347    4468 scope.go:122] "RemoveContainer" containerID="3796766161a12fe2e2179d0b7639c4ad591dd42fdd0f574001e9f9707a045a1d"
Nov 01 06:45:19.523680 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:19.523656    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt_odh-ai-gateway-infra(51e02628-a7d7-42c6-bbe9-84bb96c18c51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51"
Nov 01 06:45:19.526214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.525861    4468 generic.go:356] "Generic (PLEG): container finished" podID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerID="b5cdb5e162d1c54267c4791dc1ffc507580dc703da00db220840f80a8dc85baa" exitCode=60
Nov 01 06:45:19.526214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.525910    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" event={"ID":"6613df73-2ad6-4ef7-9623-b4499983d52e","Type":"ContainerDied","Data":"b5cdb5e162d1c54267c4791dc1ffc507580dc703da00db220840f80a8dc85baa"}
Nov 01 06:45:19.526344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.526238    4468 scope.go:122] "RemoveContainer" containerID="b5cdb5e162d1c54267c4791dc1ffc507580dc703da00db220840f80a8dc85baa"
Nov 01 06:45:19.526595 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:19.526569    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb_odh-ai-gateway-infra(6613df73-2ad6-4ef7-9623-b4499983d52e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e"
Nov 01 06:45:19.546441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.546417    4468 scope.go:122] "RemoveContainer" containerID="58f3ebaf63192d3f2b845565443980f5acb9cba2dd8c73b602832c082b3d48ff"
Nov 01 06:45:19.560753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.560659    4468 scope.go:122] "RemoveContainer" containerID="71f96900675dab63e080440dd91f88340f5c92989de17a8c09cd670c21d1b70a"
Nov 01 06:45:19.580194 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.580163    4468 scope.go:122] "RemoveContainer" containerID="33f18a2b7bcbbd2635685317989c398de8e01c84adaed04b09b0e20ff87176ef"
Nov 01 06:45:19.595747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.595675    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"]
Nov 01 06:45:19.661511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.661432    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"]
Nov 01 06:45:19.666389 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.666353    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891925-smw2n"]
Nov 01 06:45:19.690303 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.690277    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:19.724963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.724937    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:19.864290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.864267    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272\" (UniqueName: \"kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272\") pod \"f9afc4bb-a453-4f23-ac87-7a237dfae755\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") "
Nov 01 06:45:19.864494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.864343    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca\") pod \"f9afc4bb-a453-4f23-ac87-7a237dfae755\" (UID: \"f9afc4bb-a453-4f23-ac87-7a237dfae755\") "
Nov 01 06:45:19.864494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.864438    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca\") pod \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") "
Nov 01 06:45:19.864648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.864546    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9\" (UniqueName: \"kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9\") pod \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\" (UID: \"a5bcc87c-b6d6-4137-95bd-9b7f13f36494\") "
Nov 01 06:45:19.864784 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.864758    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca" pod "f9afc4bb-a453-4f23-ac87-7a237dfae755" (UID: "f9afc4bb-a453-4f23-ac87-7a237dfae755"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:19.864901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.864766    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca" pod "a5bcc87c-b6d6-4137-95bd-9b7f13f36494" (UID: "a5bcc87c-b6d6-4137-95bd-9b7f13f36494"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:19.865080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.865058    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9afc4bb-a453-4f23-ac87-7a237dfae755-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:19.865162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.865087    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:19.866648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.866629    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272" pod "f9afc4bb-a453-4f23-ac87-7a237dfae755" (UID: "f9afc4bb-a453-4f23-ac87-7a237dfae755"). InnerVolumeSpecName "kube-api-access-mt272". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:19.866723 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.866649    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9" pod "a5bcc87c-b6d6-4137-95bd-9b7f13f36494" (UID: "a5bcc87c-b6d6-4137-95bd-9b7f13f36494"). InnerVolumeSpecName "kube-api-access-4xhf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:19.882123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.882099    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"]
Nov 01 06:45:19.885926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.885899    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891925-n4v2t"]
Nov 01 06:45:19.966248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.966229    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt272\" (UniqueName: \"kubernetes.io/projected/f9afc4bb-a453-4f23-ac87-7a237dfae755-kube-api-access-mt272\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:19.966248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:19.966249    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhf9\" (UniqueName: \"kubernetes.io/projected/a5bcc87c-b6d6-4137-95bd-9b7f13f36494-kube-api-access-4xhf9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.050803 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.050774    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"]
Nov 01 06:45:20.162760 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.162687    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"]
Nov 01 06:45:20.533715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.533635    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb" event={"ID":"f9afc4bb-a453-4f23-ac87-7a237dfae755","Type":"ContainerDied","Data":"3c880473c500c241839f1ab05e35f318f8f5d588091797eec314b9bcb492ec1f"}
Nov 01 06:45:20.533715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.533656    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"
Nov 01 06:45:20.533925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.533733    4468 scope.go:122] "RemoveContainer" containerID="9128ed33cde11eace0deffd7e3c1f0bd700fc35cb62a09f046eca7bb34e17454"
Nov 01 06:45:20.535108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.535089    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"
Nov 01 06:45:20.535108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.535094    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx" event={"ID":"a5bcc87c-b6d6-4137-95bd-9b7f13f36494","Type":"ContainerDied","Data":"68662f545915b8db9aeb9175c4a4e41442151069e9214817ab295e7c1f7cf103"}
Nov 01 06:45:20.545722 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.545662    4468 scope.go:122] "RemoveContainer" containerID="4a5ac1050e98ffbe5ff0015dbbf6a887fbc0b5de0a33402afe077721954a07a0"
Nov 01 06:45:20.593735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.593678    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" path="/var/lib/kubelet/pods/98e56b16-833d-4614-b9b2-f11eb5ea7b4e/volumes"
Nov 01 06:45:20.594137 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.594120    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" path="/var/lib/kubelet/pods/b5937e71-4463-4f86-8b5f-9f78093f57d7/volumes"
Nov 01 06:45:20.676697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.676659    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"]
Nov 01 06:45:20.680447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.680421    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891925-h47bx"]
Nov 01 06:45:20.756810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.756791    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:20.758933 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.758917    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:20.761839 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.761823    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:20.787134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.787085    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"]
Nov 01 06:45:20.789813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.789792    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891925-vn5pb"]
Nov 01 06:45:20.877268 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877245    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7\" (UniqueName: \"kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7\") pod \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") "
Nov 01 06:45:20.877359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877297    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l\" (UniqueName: \"kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l\") pod \"6613df73-2ad6-4ef7-9623-b4499983d52e\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") "
Nov 01 06:45:20.877359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877326    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6\" (UniqueName: \"kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6\") pod \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") "
Nov 01 06:45:20.877359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877352    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca\") pod \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\" (UID: \"51e02628-a7d7-42c6-bbe9-84bb96c18c51\") "
Nov 01 06:45:20.877500 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877373    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca\") pod \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\" (UID: \"d3e56c70-2174-4f4c-acc6-1756dcd6fc50\") "
Nov 01 06:45:20.877555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877516    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca\") pod \"6613df73-2ad6-4ef7-9623-b4499983d52e\" (UID: \"6613df73-2ad6-4ef7-9623-b4499983d52e\") "
Nov 01 06:45:20.877784 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877757    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca" pod "51e02628-a7d7-42c6-bbe9-84bb96c18c51" (UID: "51e02628-a7d7-42c6-bbe9-84bb96c18c51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:20.877895 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877792    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca" pod "d3e56c70-2174-4f4c-acc6-1756dcd6fc50" (UID: "d3e56c70-2174-4f4c-acc6-1756dcd6fc50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:20.877957 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.877931    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca" pod "6613df73-2ad6-4ef7-9623-b4499983d52e" (UID: "6613df73-2ad6-4ef7-9623-b4499983d52e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:20.878314 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.878290    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e02628-a7d7-42c6-bbe9-84bb96c18c51-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.878384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.878319    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.878384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.878335    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6613df73-2ad6-4ef7-9623-b4499983d52e-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.879646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.879626    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6" pod "51e02628-a7d7-42c6-bbe9-84bb96c18c51" (UID: "51e02628-a7d7-42c6-bbe9-84bb96c18c51"). InnerVolumeSpecName "kube-api-access-j6bh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:20.879728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.879687    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l" pod "6613df73-2ad6-4ef7-9623-b4499983d52e" (UID: "6613df73-2ad6-4ef7-9623-b4499983d52e"). InnerVolumeSpecName "kube-api-access-c5q4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:20.879728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.879721    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7" pod "d3e56c70-2174-4f4c-acc6-1756dcd6fc50" (UID: "d3e56c70-2174-4f4c-acc6-1756dcd6fc50"). InnerVolumeSpecName "kube-api-access-kt2c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:20.979677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.979657    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2c7\" (UniqueName: \"kubernetes.io/projected/d3e56c70-2174-4f4c-acc6-1756dcd6fc50-kube-api-access-kt2c7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.979677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.979677    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5q4l\" (UniqueName: \"kubernetes.io/projected/6613df73-2ad6-4ef7-9623-b4499983d52e-kube-api-access-c5q4l\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:20.979814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:20.979686    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bh6\" (UniqueName: \"kubernetes.io/projected/51e02628-a7d7-42c6-bbe9-84bb96c18c51-kube-api-access-j6bh6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:21.553535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.553507    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"
Nov 01 06:45:21.553719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.553530    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k" event={"ID":"d3e56c70-2174-4f4c-acc6-1756dcd6fc50","Type":"ContainerDied","Data":"a023f9853c5ed176dbea9df017621c60da426b748d9e2ad3cd7028b77f8b8910"}
Nov 01 06:45:21.553719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.553595    4468 scope.go:122] "RemoveContainer" containerID="d1ff3ccaa9183f4477efc9d1daa52b265303f74a86e1382448f0f0da7a41ebe7"
Nov 01 06:45:21.555015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.554994    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"
Nov 01 06:45:21.555299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.555013    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt" event={"ID":"51e02628-a7d7-42c6-bbe9-84bb96c18c51","Type":"ContainerDied","Data":"256b25ebd3588f9b1bd5a1b565f6db9f0413e23eb0c8d2739e3be0d805836603"}
Nov 01 06:45:21.562096 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.562074    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"
Nov 01 06:45:21.562259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.562096    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb" event={"ID":"6613df73-2ad6-4ef7-9623-b4499983d52e","Type":"ContainerDied","Data":"741564678dc0903574b3def86f1d6e0ec2d6f2fa77671db522f16f56bf85a0c9"}
Nov 01 06:45:21.566595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.566566    4468 scope.go:122] "RemoveContainer" containerID="3796766161a12fe2e2179d0b7639c4ad591dd42fdd0f574001e9f9707a045a1d"
Nov 01 06:45:21.575961 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.575943    4468 scope.go:122] "RemoveContainer" containerID="b5cdb5e162d1c54267c4791dc1ffc507580dc703da00db220840f80a8dc85baa"
Nov 01 06:45:21.720294 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.720263    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"]
Nov 01 06:45:21.724354 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.724330    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891925-srr7k"]
Nov 01 06:45:21.830833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.830770    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"]
Nov 01 06:45:21.834711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.834692    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891925-wfwlb"]
Nov 01 06:45:21.943114 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.943087    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"]
Nov 01 06:45:21.946426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:21.946404    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891925-6gbbt"]
Nov 01 06:45:22.068102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068075    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"]
Nov 01 06:45:22.068583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068571    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068585    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068595    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068601    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068617    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068623    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068629    4468 cpu_manager.go:414] "containerMap: removing container" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068633    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:22.068640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068643    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068647    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068656    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068661    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068667    4468 cpu_manager.go:414] "containerMap: removing container" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068671    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068677    4468 cpu_manager.go:414] "containerMap: removing container" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068681    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068688    4468 cpu_manager.go:414] "containerMap: removing container" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068692    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068699    4468 cpu_manager.go:414] "containerMap: removing container" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068702    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068712    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068716    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068726    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068730    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068738    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068742    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068751    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068755    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068761    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068765    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068775    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068779    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068787    4468 cpu_manager.go:414] "containerMap: removing container" podUID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" containerName="main"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068792    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" containerName="main"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068858    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068866    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:22.068865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068873    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068879    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cdcf27-c4f1-4679-a8ec-d2e9ea73da47" containerName="main"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068885    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068890    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068897    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068902    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068907    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068911    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068916    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068921    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068926    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068931    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068937    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068943    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068949    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068953    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:22.069648 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.068957    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:22.073538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.073521    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.091719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.091667    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.091810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.091725    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86pm4\" (UniqueName: \"kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.091851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.091809    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.126215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.126190    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:45:22.126345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.126217    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:45:22.126345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.126278    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:45:22.126345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.126282    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:45:22.128752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.128731    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"]
Nov 01 06:45:22.193216 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.193194    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.193364 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.193235    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86pm4\" (UniqueName: \"kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.193364 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.193267    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.193918 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.193901    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.195769 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.195748    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.249014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.248986    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86pm4\" (UniqueName: \"kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4\") pod \"maas-api-revoke-keys-e2e-derive-030936cd-x6wf5\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.386456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.386391    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:22.598245 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.598212    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e02628-a7d7-42c6-bbe9-84bb96c18c51" path="/var/lib/kubelet/pods/51e02628-a7d7-42c6-bbe9-84bb96c18c51/volumes"
Nov 01 06:45:22.598615 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.598601    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" path="/var/lib/kubelet/pods/6613df73-2ad6-4ef7-9623-b4499983d52e/volumes"
Nov 01 06:45:22.598942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.598930    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" path="/var/lib/kubelet/pods/a5bcc87c-b6d6-4137-95bd-9b7f13f36494/volumes"
Nov 01 06:45:22.599569 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.599554    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" path="/var/lib/kubelet/pods/d3e56c70-2174-4f4c-acc6-1756dcd6fc50/volumes"
Nov 01 06:45:22.599901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.599890    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" path="/var/lib/kubelet/pods/f9afc4bb-a453-4f23-ac87-7a237dfae755/volumes"
Nov 01 06:45:22.758405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:22.758376    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"]
Nov 01 06:45:22.759298 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:22.759270    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354bc19f_61e0_420c_8181_76c69f7d9961.slice/crio-46631f75a4133d5b79a84ed7b0d9a7734cb7478a0d9a57a81b13250e099d5847 WatchSource:0}: Error finding container 46631f75a4133d5b79a84ed7b0d9a7734cb7478a0d9a57a81b13250e099d5847: Status 404 returned error can't find the container with id 46631f75a4133d5b79a84ed7b0d9a7734cb7478a0d9a57a81b13250e099d5847
Nov 01 06:45:23.582765 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:23.582733    4468 generic.go:356] "Generic (PLEG): container finished" podID="354bc19f-61e0-420c-8181-76c69f7d9961" containerID="dfa48a4894c11bfa4d431de6f3597ab41bace52350ed74210ac69a216a0d9f22" exitCode=0
Nov 01 06:45:23.583078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:23.582811    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5" event={"ID":"354bc19f-61e0-420c-8181-76c69f7d9961","Type":"ContainerDied","Data":"dfa48a4894c11bfa4d431de6f3597ab41bace52350ed74210ac69a216a0d9f22"}
Nov 01 06:45:23.583078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:23.582863    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5" event={"ID":"354bc19f-61e0-420c-8181-76c69f7d9961","Type":"ContainerStarted","Data":"46631f75a4133d5b79a84ed7b0d9a7734cb7478a0d9a57a81b13250e099d5847"}
Nov 01 06:45:24.733193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.733169    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:24.822915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.822885    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca\") pod \"354bc19f-61e0-420c-8181-76c69f7d9961\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") "
Nov 01 06:45:24.823045 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.822953    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca\") pod \"354bc19f-61e0-420c-8181-76c69f7d9961\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") "
Nov 01 06:45:24.823045 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.822974    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4\" (UniqueName: \"kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4\") pod \"354bc19f-61e0-420c-8181-76c69f7d9961\" (UID: \"354bc19f-61e0-420c-8181-76c69f7d9961\") "
Nov 01 06:45:24.823308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.823280    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca" pod "354bc19f-61e0-420c-8181-76c69f7d9961" (UID: "354bc19f-61e0-420c-8181-76c69f7d9961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:24.825102 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.825074    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca" pod "354bc19f-61e0-420c-8181-76c69f7d9961" (UID: "354bc19f-61e0-420c-8181-76c69f7d9961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:45:24.825178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.825079    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4" pod "354bc19f-61e0-420c-8181-76c69f7d9961" (UID: "354bc19f-61e0-420c-8181-76c69f7d9961"). InnerVolumeSpecName "kube-api-access-86pm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:24.924505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.924405    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:24.924505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.924432    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86pm4\" (UniqueName: \"kubernetes.io/projected/354bc19f-61e0-420c-8181-76c69f7d9961-kube-api-access-86pm4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:24.924505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:24.924444    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354bc19f-61e0-420c-8181-76c69f7d9961-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:25.369403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:25.369369    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:45:25.597039 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:25.597014    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"
Nov 01 06:45:25.597198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:25.597013    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5" event={"ID":"354bc19f-61e0-420c-8181-76c69f7d9961","Type":"ContainerDied","Data":"46631f75a4133d5b79a84ed7b0d9a7734cb7478a0d9a57a81b13250e099d5847"}
Nov 01 06:45:25.597198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:25.597129    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46631f75a4133d5b79a84ed7b0d9a7734cb7478a0d9a57a81b13250e099d5847"
Nov 01 06:45:32.947531 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:32.947460    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:45:34.534360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.534318    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"]
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536140    4468 cpu_manager.go:414] "containerMap: removing container" podUID="354bc19f-61e0-420c-8181-76c69f7d9961" containerName="revoke-keys"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536167    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354bc19f-61e0-420c-8181-76c69f7d9961" containerName="revoke-keys"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536189    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536197    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536235    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536248    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bcc87c-b6d6-4137-95bd-9b7f13f36494" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536324    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536331    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6613df73-2ad6-4ef7-9623-b4499983d52e" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536356    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536363    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5937e71-4463-4f86-8b5f-9f78093f57d7" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536390    4468 cpu_manager.go:414] "containerMap: removing container" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536396    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536677    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9afc4bb-a453-4f23-ac87-7a237dfae755" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536712    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="354bc19f-61e0-420c-8181-76c69f7d9961" containerName="revoke-keys"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536723    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e56c70-2174-4f4c-acc6-1756dcd6fc50" containerName="cleanup"
Nov 01 06:45:34.539515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.536738    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e56b16-833d-4614-b9b2-f11eb5ea7b4e" containerName="cleanup"
Nov 01 06:45:34.542898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.542872    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.598103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.598072    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"]
Nov 01 06:45:34.643685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.643651    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-47bc8f48-openshift-default-dockercfg-dxng8\""
Nov 01 06:45:34.647514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.647487    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"]
Nov 01 06:45:34.647750 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.647724    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" containerID="cri-o://e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83" gracePeriod=30
Nov 01 06:45:34.671033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.670942    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671109    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671193 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671185    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671223    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671311 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671252    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxswb\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671317    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671349    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671401    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.671601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.671430    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.772923 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.772888    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.772962    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.772994    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773023    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxswb\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773078    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773140    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773188    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773214    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773631    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.773707 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773661    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.774127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.773751    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.774127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.774049    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.774217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.774164    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.774499 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.774459    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.775665 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.775644    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.775863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.775846    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.793731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.793675    4468 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:45:34.793810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.793726    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:45:34.829515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.829493    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.829638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.829570    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxswb\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb\") pod \"e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") " pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:34.863974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:34.860760    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:35.310834 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.310802    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"]
Nov 01 06:45:35.311981 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:45:35.311956    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d705c95_3058_430e_8915_ef05b0bb6175.slice/crio-f9d48311cd1e99f1ba539dad87728cd09fc67ab41ec0bef4cac226a002cfa9b9 WatchSource:0}: Error finding container f9d48311cd1e99f1ba539dad87728cd09fc67ab41ec0bef4cac226a002cfa9b9: Status 404 returned error can't find the container with id f9d48311cd1e99f1ba539dad87728cd09fc67ab41ec0bef4cac226a002cfa9b9
Nov 01 06:45:35.314568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.314537    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:45:35.314662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.314612    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:45:35.314662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.314643    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:45:35.663438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.663350    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" event={"ID":"4d705c95-3058-430e-8915-ef05b0bb6175","Type":"ContainerStarted","Data":"37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"}
Nov 01 06:45:35.663438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.663411    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" event={"ID":"4d705c95-3058-430e-8915-ef05b0bb6175","Type":"ContainerStarted","Data":"f9d48311cd1e99f1ba539dad87728cd09fc67ab41ec0bef4cac226a002cfa9b9"}
Nov 01 06:45:35.780718 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.780667    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podStartSLOduration=1.780653144 podStartE2EDuration="1.780653144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:35.777805065 +0000 UTC m=+2147.345611918" watchObservedRunningTime="2026-11-01 06:45:35.780653144 +0000 UTC m=+2147.348459972"
Nov 01 06:45:35.861430 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:35.861401    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:36.866884 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:36.866853    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:37.675266 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:37.675240    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:37.676306 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:37.676287    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:45:39.896864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.896838    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:45:39.971442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971362    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971415    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971454    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971519    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971645    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971803    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971817    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:39.971879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971831    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:39.971969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971893    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.971969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.971936    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.972231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.972077    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs\") pod \"9a4382c7-7f1c-4274-914c-a2a106096b52\" (UID: \"9a4382c7-7f1c-4274-914c-a2a106096b52\") "
Nov 01 06:45:39.972231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.972156    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:45:39.972446 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.972403    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:39.972446 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.972422    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:39.972984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.972960    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:39.973081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.972989    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:39.973081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.973005    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:39.973081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.973020    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:39.973081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.973034    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4382c7-7f1c-4274-914c-a2a106096b52-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:39.974404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.974352    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:45:39.974666 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.974639    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:45:39.974873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.974849    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:39.975113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:39.975097    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6" pod "9a4382c7-7f1c-4274-914c-a2a106096b52" (UID: "9a4382c7-7f1c-4274-914c-a2a106096b52"). InnerVolumeSpecName "kube-api-access-v8wz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:45:40.074365 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.074335    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:40.074365 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.074362    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:40.074592 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.074373    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wz6\" (UniqueName: \"kubernetes.io/projected/9a4382c7-7f1c-4274-914c-a2a106096b52-kube-api-access-v8wz6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:40.074592 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.074383    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4382c7-7f1c-4274-914c-a2a106096b52-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:45:40.695990 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.695956    4468 generic.go:356] "Generic (PLEG): container finished" podID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerID="e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83" exitCode=0
Nov 01 06:45:40.696157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.695995    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" event={"ID":"9a4382c7-7f1c-4274-914c-a2a106096b52","Type":"ContainerDied","Data":"e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83"}
Nov 01 06:45:40.696157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.696004    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"
Nov 01 06:45:40.696157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.696043    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp" event={"ID":"9a4382c7-7f1c-4274-914c-a2a106096b52","Type":"ContainerDied","Data":"7e1502d4f120d847774f667522c15d6888e52f236ab06c475d0e8b8084c1d0b6"}
Nov 01 06:45:40.696157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.696072    4468 scope.go:122] "RemoveContainer" containerID="e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83"
Nov 01 06:45:40.705936 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.705912    4468 scope.go:122] "RemoveContainer" containerID="e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83"
Nov 01 06:45:40.706215 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:45:40.706192    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83\": container with ID starting with e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83 not found: ID does not exist" containerID="e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83"
Nov 01 06:45:40.706280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.706228    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83"} err="failed to get container status \"e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83\": rpc error: code = NotFound desc = could not find container \"e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83\": container with ID starting with e44d627989bc65ad4368a9858a1b96179989393a6784392367a9ee41a9b07e83 not found: ID does not exist"
Nov 01 06:45:40.859033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.858934    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"]
Nov 01 06:45:40.908419 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:40.908387    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-bxfjp"]
Nov 01 06:45:42.590084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:42.590055    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" path="/var/lib/kubelet/pods/9a4382c7-7f1c-4274-914c-a2a106096b52/volumes"
Nov 01 06:45:51.035636 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:45:51.035604    4468 scope.go:122] "RemoveContainer" containerID="cacddbb88bd3f210696163637f1ead4fe12fab5c4433574b5b22ec9115d89b4c"
Nov 01 06:46:02.947128 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:46:02.947090    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:46:08.183525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:08.183482    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:46:14.378901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:14.378868    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"]
Nov 01 06:46:14.428614 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:14.428535    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-030936cd-x6wf5"]
Nov 01 06:46:14.472572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:14.472538    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"]
Nov 01 06:46:14.472821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:14.472795    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy" containerID="cri-o://3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7" gracePeriod=30
Nov 01 06:46:14.607110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:14.607073    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354bc19f-61e0-420c-8181-76c69f7d9961" path="/var/lib/kubelet/pods/354bc19f-61e0-420c-8181-76c69f7d9961/volumes"
Nov 01 06:46:16.249704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.249661    4468 patch_prober.go:29] interesting pod/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:46:16.250114 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.249753    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:46:16.858254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.858218    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"]
Nov 01 06:46:16.858807 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.858792    4468 cpu_manager.go:414] "containerMap: removing container" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy"
Nov 01 06:46:16.858885 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.858809    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy"
Nov 01 06:46:16.858983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.858969    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4382c7-7f1c-4274-914c-a2a106096b52" containerName="istio-proxy"
Nov 01 06:46:16.862533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.862514    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.965525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.965488    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"]
Nov 01 06:46:16.966915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.966888    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.966944    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967098    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967157    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967205 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967186    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967211    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btg9r\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967292 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967279    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967396 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967381    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:16.967441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:16.967414    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.009320 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.009288    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-caf74d-openshift-default-dockercfg-thdc5\""
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068495    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068570    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068601    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068635    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btg9r\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068718    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068820    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068853    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068887    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.070538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.068930    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.074258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.074220    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.074877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.074837    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.078525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.075938    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.078525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.076205    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.078525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.077672    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.078525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.077862    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.084613 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.084039    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.130085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.130007    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btg9r\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.130381 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.130359    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token\") pod \"e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") " pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.176010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.175977    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:17.607842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.607710    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"]
Nov 01 06:46:17.609933 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:46:17.609901    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f86d36a_3c78_44bd_bf20_00e260f66c1e.slice/crio-c9b7a57e62e66251d7a4b8c62dfd03aa75919211897960c43a3a5ee77f3cf304 WatchSource:0}: Error finding container c9b7a57e62e66251d7a4b8c62dfd03aa75919211897960c43a3a5ee77f3cf304: Status 404 returned error can't find the container with id c9b7a57e62e66251d7a4b8c62dfd03aa75919211897960c43a3a5ee77f3cf304
Nov 01 06:46:17.612669 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.612636    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:17.612768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.612704    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:17.612768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.612733    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:17.930028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.929937    4468 generic.go:356] "Generic (PLEG): container finished" podID="be32ed1d-772c-4a98-926a-653cb28329c1" containerID="7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8" exitCode=0
Nov 01 06:46:17.930028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.930002    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" event={"ID":"be32ed1d-772c-4a98-926a-653cb28329c1","Type":"ContainerDied","Data":"7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"}
Nov 01 06:46:17.930561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.930543    4468 scope.go:122] "RemoveContainer" containerID="7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"
Nov 01 06:46:17.932001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.931961    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" event={"ID":"6f86d36a-3c78-44bd-bf20-00e260f66c1e","Type":"ContainerStarted","Data":"84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"}
Nov 01 06:46:17.932116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.932027    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" event={"ID":"6f86d36a-3c78-44bd-bf20-00e260f66c1e","Type":"ContainerStarted","Data":"c9b7a57e62e66251d7a4b8c62dfd03aa75919211897960c43a3a5ee77f3cf304"}
Nov 01 06:46:17.932219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.932195    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:17.932279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.932263    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:17.932330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:17.932297    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:46:18.162255 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:18.162203    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podStartSLOduration=2.162188716 podStartE2EDuration="2.162188716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:18.159505131 +0000 UTC m=+2189.727311962" watchObservedRunningTime="2026-11-01 06:46:18.162188716 +0000 UTC m=+2189.729995665"
Nov 01 06:46:18.176838 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:18.176812    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:18.183016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:18.182956    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:18.940310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:18.940276    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" event={"ID":"be32ed1d-772c-4a98-926a-653cb28329c1","Type":"ContainerStarted","Data":"858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b"}
Nov 01 06:46:18.940916 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:18.940898    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:18.941858 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:18.941839    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:46:19.591581 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.591550    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:46:19.591762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.591738    4468 patch_prober.go:29] interesting pod/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:46:19.591827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.591802    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 06:46:19.771598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.771569    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:46:19.842842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.842804    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.842890    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.842912    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.842942    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.842975    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843030 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843023    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843056    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843150    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843179    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket\") pod \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\" (UID: \"49078d4c-aabb-44cf-bea4-6b9a67de3f8b\") "
Nov 01 06:46:19.843366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843229    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:19.843366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843255    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:19.843366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843348    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:46:19.843532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843443    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:19.843694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843666    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:19.843982 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843730    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.843982 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843745    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.843982 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843755    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.843982 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.843763    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.845490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.845416    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:19.845588 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.845505    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:46:19.846059 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.845697    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "kube-api-access-b6bkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:19.846059 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.845723    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy" pod "49078d4c-aabb-44cf-bea4-6b9a67de3f8b" (UID: "49078d4c-aabb-44cf-bea4-6b9a67de3f8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:19.945165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.945121    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bkw\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-kube-api-access-b6bkw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.945165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.945159    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.945165 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.945175    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.945808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.945188    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.945808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.945203    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49078d4c-aabb-44cf-bea4-6b9a67de3f8b-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:19.948043 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.948008    4468 generic.go:356] "Generic (PLEG): container finished" podID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerID="3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7" exitCode=0
Nov 01 06:46:19.948188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.948062    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"
Nov 01 06:46:19.948188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.948082    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" event={"ID":"49078d4c-aabb-44cf-bea4-6b9a67de3f8b","Type":"ContainerDied","Data":"3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7"}
Nov 01 06:46:19.948188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.948131    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89" event={"ID":"49078d4c-aabb-44cf-bea4-6b9a67de3f8b","Type":"ContainerDied","Data":"9a00b41a04d1f6d754b75a7a63da01afbdd498f5254729ea83efbe7f213b9aac"}
Nov 01 06:46:19.948188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.948154    4468 scope.go:122] "RemoveContainer" containerID="3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7"
Nov 01 06:46:19.959770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.959749    4468 scope.go:122] "RemoveContainer" containerID="7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"
Nov 01 06:46:19.969181 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.969162    4468 scope.go:122] "RemoveContainer" containerID="3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7"
Nov 01 06:46:19.969488 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:46:19.969450    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7\": container with ID starting with 3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7 not found: ID does not exist" containerID="3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7"
Nov 01 06:46:19.969536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.969502    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7"} err="failed to get container status \"3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7\": rpc error: code = NotFound desc = could not find container \"3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7\": container with ID starting with 3b5ea196296ccf9f2f84ef816a4e321a5e92a60393a93a87df0af526f3f087a7 not found: ID does not exist"
Nov 01 06:46:19.969536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.969522    4468 scope.go:122] "RemoveContainer" containerID="7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"
Nov 01 06:46:19.969764 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:46:19.969750    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673\": container with ID starting with 7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673 not found: ID does not exist" containerID="7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"
Nov 01 06:46:19.969802 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:19.969768    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673"} err="failed to get container status \"7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673\": rpc error: code = NotFound desc = could not find container \"7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673\": container with ID starting with 7875ac48c09985e2ac9e531074f70689a60960f52d1214abe7d74c331134c673 not found: ID does not exist"
Nov 01 06:46:20.130631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:20.130545    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"]
Nov 01 06:46:20.133141 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:20.133111    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-030936cd-openshift-default-55f6b8f4fc-mzn89"]
Nov 01 06:46:20.589813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:20.589776    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" path="/var/lib/kubelet/pods/49078d4c-aabb-44cf-bea4-6b9a67de3f8b/volumes"
Nov 01 06:46:20.591735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:20.591708    4468 patch_prober.go:29] interesting pod/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:46:20.591868 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:20.591775    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 06:46:21.591181 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:21.591143    4468 patch_prober.go:29] interesting pod/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:46:21.591643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:21.591196    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 06:46:22.590867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:22.590830    4468 patch_prober.go:29] interesting pod/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:46:22.591031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:22.590891    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 06:46:23.590948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:23.590915    4468 patch_prober.go:29] interesting pod/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 01 06:46:23.591318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:23.590974    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 01 06:46:24.595446 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:24.595418    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:46:24.595932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:24.595734    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:46:24.596528 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:24.596510    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:46:31.238984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:31.238907    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:46:32.947140 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:46:32.947106    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:46:55.007533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:55.007497    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:46:59.530046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.530010    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"]
Nov 01 06:46:59.531843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.531819    4468 cpu_manager.go:414] "containerMap: removing container" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy"
Nov 01 06:46:59.531843 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.531842    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy"
Nov 01 06:46:59.532055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.532003    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy"
Nov 01 06:46:59.537014 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.536995    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.596646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.596614    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d855db-openshift-default-dockercfg-bcn5l\""
Nov 01 06:46:59.668273 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.668230    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.668633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.668608    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.668805 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.668787    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.668934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.668914    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxdw\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.669060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.669045    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.669162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.669145    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.669266 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.669253    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.669591 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.669567    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.669770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.669750    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.700901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.700863    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"]
Nov 01 06:46:59.770751 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770714    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.770927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770784    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.770927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770805    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxdw\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.770927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770827    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.770927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770849    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.770927 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770879    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.771126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770946    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.771126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.770994    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.771126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.771103    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.772429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.771717    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.772429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.772091    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.772429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.772228    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.772429 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.772387    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.773005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.772980    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.773515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.773495    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.774270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.774249    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.829905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.829871    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxdw\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.830062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.829874    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token\") pod \"e2e-api-b-d855db-openshift-default-577cbd8699-6djwm\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") " pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:46:59.852842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:46:59.852815    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:47:00.331805 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:00.331646    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"]
Nov 01 06:47:00.360720 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:00.360692    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8161e4f0_456c_427f_ae79_b2fa97b5ea6d.slice/crio-695a89c45d058374ccbc11fd1c1fd11add7f23551bc2f009562dea084b209a51 WatchSource:0}: Error finding container 695a89c45d058374ccbc11fd1c1fd11add7f23551bc2f009562dea084b209a51: Status 404 returned error can't find the container with id 695a89c45d058374ccbc11fd1c1fd11add7f23551bc2f009562dea084b209a51
Nov 01 06:47:00.363366 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:00.363333    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:00.363510 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:00.363404    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:00.363510 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:00.363436    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:01.200297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:01.200252    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" event={"ID":"8161e4f0-456c-427f-ae79-b2fa97b5ea6d","Type":"ContainerStarted","Data":"857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"}
Nov 01 06:47:01.200745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:01.200322    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" event={"ID":"8161e4f0-456c-427f-ae79-b2fa97b5ea6d","Type":"ContainerStarted","Data":"695a89c45d058374ccbc11fd1c1fd11add7f23551bc2f009562dea084b209a51"}
Nov 01 06:47:01.366137 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:01.366079    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podStartSLOduration=2.366059521 podStartE2EDuration="2.366059521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:01.36354425 +0000 UTC m=+2232.931351080" watchObservedRunningTime="2026-11-01 06:47:01.366059521 +0000 UTC m=+2232.933866353"
Nov 01 06:47:01.853742 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:01.853707    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:47:01.860054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:01.860028    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:47:02.206736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:02.206665    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:47:02.207581 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:02.207563    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:47:02.949241 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:02.949205    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:47:04.219488 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:04.219379    4468 generic.go:356] "Generic (PLEG): container finished" podID="4d705c95-3058-430e-8915-ef05b0bb6175" containerID="37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165" exitCode=0
Nov 01 06:47:04.219488 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:04.219457    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" event={"ID":"4d705c95-3058-430e-8915-ef05b0bb6175","Type":"ContainerDied","Data":"37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"}
Nov 01 06:47:04.220075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:04.220057    4468 scope.go:122] "RemoveContainer" containerID="37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"
Nov 01 06:47:04.221747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:04.221713    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:04.221825 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:04.221809    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:04.221879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:04.221861    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:05.233350 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:05.233313    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" event={"ID":"4d705c95-3058-430e-8915-ef05b0bb6175","Type":"ContainerStarted","Data":"777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49"}
Nov 01 06:47:05.861158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:05.861124    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:47:05.862185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:05.862158    4468 patch_prober.go:29] interesting pod/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:47:05.862315 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:05.862222    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:47:06.580058 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.580024    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"]
Nov 01 06:47:06.580601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.580586    4468 cpu_manager.go:414] "containerMap: removing container" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy"
Nov 01 06:47:06.580666 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.580602    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy"
Nov 01 06:47:06.580708 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.580702    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="49078d4c-aabb-44cf-bea4-6b9a67de3f8b" containerName="istio-proxy"
Nov 01 06:47:06.583221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.583204    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.654075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.654041    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.654249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.654209    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hp4q\" (UniqueName: \"kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.654355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.654341    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.686626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.686593    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:47:06.686626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.686609    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:47:06.686626 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.686610    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:47:06.686867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.686655    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:47:06.689934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.689913    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"]
Nov 01 06:47:06.755453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.755418    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hp4q\" (UniqueName: \"kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.755644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.755542    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.755644 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.755581    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.756179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.756159    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.758044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.758021    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.828111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.828066    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hp4q\" (UniqueName: \"kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q\") pod \"maas-api-revoke-sub-89b919e0-9lp42\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:06.861447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.861373    4468 patch_prober.go:29] interesting pod/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:47:06.861447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.861422    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:47:06.895685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:06.895646    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:07.297415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:07.297389    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"]
Nov 01 06:47:07.298275 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:07.298245    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60a1dd4_4405_4e7d_9b33_4b15513682bb.slice/crio-d0aeea98ff3b5c6d727e3f9b67e6c1478e098ffd0b2c4d49ef87c16b19dea2e3 WatchSource:0}: Error finding container d0aeea98ff3b5c6d727e3f9b67e6c1478e098ffd0b2c4d49ef87c16b19dea2e3: Status 404 returned error can't find the container with id d0aeea98ff3b5c6d727e3f9b67e6c1478e098ffd0b2c4d49ef87c16b19dea2e3
Nov 01 06:47:07.675557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:07.675530    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:47:07.861824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:07.861740    4468 patch_prober.go:29] interesting pod/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:47:07.861824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:07.861796    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:47:08.253638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:08.253552    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42" event={"ID":"a60a1dd4-4405-4e7d-9b33-4b15513682bb","Type":"ContainerStarted","Data":"db9fff4abc0733210756cd2e080cfe3e4662f16df3dfbabaf1c178d751c60aff"}
Nov 01 06:47:08.253638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:08.253601    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42" event={"ID":"a60a1dd4-4405-4e7d-9b33-4b15513682bb","Type":"ContainerStarted","Data":"d0aeea98ff3b5c6d727e3f9b67e6c1478e098ffd0b2c4d49ef87c16b19dea2e3"}
Nov 01 06:47:08.389215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:08.389164    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42" podStartSLOduration=2.38914518 podStartE2EDuration="2.38914518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:08.387803794 +0000 UTC m=+2239.955610638" watchObservedRunningTime="2026-11-01 06:47:08.38914518 +0000 UTC m=+2239.956952055"
Nov 01 06:47:08.861385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:08.861296    4468 patch_prober.go:29] interesting pod/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:47:08.861385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:08.861348    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:47:09.261773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:09.261685    4468 generic.go:356] "Generic (PLEG): container finished" podID="a60a1dd4-4405-4e7d-9b33-4b15513682bb" containerID="db9fff4abc0733210756cd2e080cfe3e4662f16df3dfbabaf1c178d751c60aff" exitCode=0
Nov 01 06:47:09.261773 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:09.261734    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42" event={"ID":"a60a1dd4-4405-4e7d-9b33-4b15513682bb","Type":"ContainerDied","Data":"db9fff4abc0733210756cd2e080cfe3e4662f16df3dfbabaf1c178d751c60aff"}
Nov 01 06:47:09.861325 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:09.861286    4468 patch_prober.go:29] interesting pod/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:47:09.861533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:09.861348    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:47:10.426525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.426496    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:10.498515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.498483    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca\") pod \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") "
Nov 01 06:47:10.498711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.498529    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q\" (UniqueName: \"kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q\") pod \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") "
Nov 01 06:47:10.498711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.498568    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca\") pod \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\" (UID: \"a60a1dd4-4405-4e7d-9b33-4b15513682bb\") "
Nov 01 06:47:10.498951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.498927    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca" pod "a60a1dd4-4405-4e7d-9b33-4b15513682bb" (UID: "a60a1dd4-4405-4e7d-9b33-4b15513682bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:10.499045 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.499027    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:10.500788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.500757    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q" pod "a60a1dd4-4405-4e7d-9b33-4b15513682bb" (UID: "a60a1dd4-4405-4e7d-9b33-4b15513682bb"). InnerVolumeSpecName "kube-api-access-4hp4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:10.500899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.500830    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca" pod "a60a1dd4-4405-4e7d-9b33-4b15513682bb" (UID: "a60a1dd4-4405-4e7d-9b33-4b15513682bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:10.600212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.600179    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hp4q\" (UniqueName: \"kubernetes.io/projected/a60a1dd4-4405-4e7d-9b33-4b15513682bb-kube-api-access-4hp4q\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:10.600212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.600208    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a1dd4-4405-4e7d-9b33-4b15513682bb-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:10.639001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.638970    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"]
Nov 01 06:47:10.639547 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.639533    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a60a1dd4-4405-4e7d-9b33-4b15513682bb" containerName="revoke-keys"
Nov 01 06:47:10.639595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.639549    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60a1dd4-4405-4e7d-9b33-4b15513682bb" containerName="revoke-keys"
Nov 01 06:47:10.639672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.639663    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60a1dd4-4405-4e7d-9b33-4b15513682bb" containerName="revoke-keys"
Nov 01 06:47:10.642103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.642082    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.700844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.700808    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"]
Nov 01 06:47:10.701396 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.701373    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.701561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.701528    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.701771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.701751    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlk29\" (UniqueName: \"kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.802783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.802750    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlk29\" (UniqueName: \"kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.802966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.802817    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.802966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.802873    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.803623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.803598    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.805234 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.805199    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.858900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.858828    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlk29\" (UniqueName: \"kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29\") pod \"maas-api-revoke-sub-ebe5d85e-m5gqd\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:10.866398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.866365    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:47:10.867335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.867321    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:47:10.955185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:10.955151    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:11.290737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:11.288768    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"
Nov 01 06:47:11.290737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:11.288879    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42" event={"ID":"a60a1dd4-4405-4e7d-9b33-4b15513682bb","Type":"ContainerDied","Data":"d0aeea98ff3b5c6d727e3f9b67e6c1478e098ffd0b2c4d49ef87c16b19dea2e3"}
Nov 01 06:47:11.290737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:11.288934    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0aeea98ff3b5c6d727e3f9b67e6c1478e098ffd0b2c4d49ef87c16b19dea2e3"
Nov 01 06:47:11.328540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:11.328511    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"]
Nov 01 06:47:11.330101 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:11.330072    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852ac969_e217_4fb0_9799_ecafbf65ee93.slice/crio-9fb94282b9b7eb0b7fdd9cd8dde76e6d7a1017abfe51e66b35d8d5e41f29eead WatchSource:0}: Error finding container 9fb94282b9b7eb0b7fdd9cd8dde76e6d7a1017abfe51e66b35d8d5e41f29eead: Status 404 returned error can't find the container with id 9fb94282b9b7eb0b7fdd9cd8dde76e6d7a1017abfe51e66b35d8d5e41f29eead
Nov 01 06:47:12.296734 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:12.296702    4468 generic.go:356] "Generic (PLEG): container finished" podID="852ac969-e217-4fb0-9799-ecafbf65ee93" containerID="8b675135e75ef73e0de88de1cd1a188ec3c287dde04e5b08d3ea9e97c784b621" exitCode=0
Nov 01 06:47:12.297127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:12.296774    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd" event={"ID":"852ac969-e217-4fb0-9799-ecafbf65ee93","Type":"ContainerDied","Data":"8b675135e75ef73e0de88de1cd1a188ec3c287dde04e5b08d3ea9e97c784b621"}
Nov 01 06:47:12.297127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:12.296809    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd" event={"ID":"852ac969-e217-4fb0-9799-ecafbf65ee93","Type":"ContainerStarted","Data":"9fb94282b9b7eb0b7fdd9cd8dde76e6d7a1017abfe51e66b35d8d5e41f29eead"}
Nov 01 06:47:13.459769 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.459743    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:13.577568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.577489    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29\" (UniqueName: \"kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29\") pod \"852ac969-e217-4fb0-9799-ecafbf65ee93\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") "
Nov 01 06:47:13.577729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.577606    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca\") pod \"852ac969-e217-4fb0-9799-ecafbf65ee93\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") "
Nov 01 06:47:13.577729 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.577677    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca\") pod \"852ac969-e217-4fb0-9799-ecafbf65ee93\" (UID: \"852ac969-e217-4fb0-9799-ecafbf65ee93\") "
Nov 01 06:47:13.578084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.577984    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca" pod "852ac969-e217-4fb0-9799-ecafbf65ee93" (UID: "852ac969-e217-4fb0-9799-ecafbf65ee93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:13.578199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.578098    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:13.580204 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.580176    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29" pod "852ac969-e217-4fb0-9799-ecafbf65ee93" (UID: "852ac969-e217-4fb0-9799-ecafbf65ee93"). InnerVolumeSpecName "kube-api-access-nlk29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:13.580324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.580224    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca" pod "852ac969-e217-4fb0-9799-ecafbf65ee93" (UID: "852ac969-e217-4fb0-9799-ecafbf65ee93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:13.679796 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.679760    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlk29\" (UniqueName: \"kubernetes.io/projected/852ac969-e217-4fb0-9799-ecafbf65ee93-kube-api-access-nlk29\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:13.679955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:13.679812    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ac969-e217-4fb0-9799-ecafbf65ee93-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:14.320594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:14.320564    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"
Nov 01 06:47:14.320594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:14.320587    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd" event={"ID":"852ac969-e217-4fb0-9799-ecafbf65ee93","Type":"ContainerDied","Data":"9fb94282b9b7eb0b7fdd9cd8dde76e6d7a1017abfe51e66b35d8d5e41f29eead"}
Nov 01 06:47:14.320801 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:14.320635    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb94282b9b7eb0b7fdd9cd8dde76e6d7a1017abfe51e66b35d8d5e41f29eead"
Nov 01 06:47:15.045110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:15.045077    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"]
Nov 01 06:47:15.094394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:15.094368    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b0eb1a-q88nv"]
Nov 01 06:47:16.590029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:16.589998    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbce2f4e-6fda-439b-8b40-416cd700cb98" path="/var/lib/kubelet/pods/dbce2f4e-6fda-439b-8b40-416cd700cb98/volumes"
Nov 01 06:47:18.295783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.295742    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"]
Nov 01 06:47:18.296530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.296510    4468 cpu_manager.go:414] "containerMap: removing container" podUID="852ac969-e217-4fb0-9799-ecafbf65ee93" containerName="revoke-keys"
Nov 01 06:47:18.296530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.296529    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852ac969-e217-4fb0-9799-ecafbf65ee93" containerName="revoke-keys"
Nov 01 06:47:18.296697 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.296679    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="852ac969-e217-4fb0-9799-ecafbf65ee93" containerName="revoke-keys"
Nov 01 06:47:18.300296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.300275    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.357847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.357817    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"]
Nov 01 06:47:18.402689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.402654    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:47:18.402689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.402654    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:47:18.402915 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.402665    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:47:18.403431 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.403417    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:47:18.436300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.436275    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.436415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.436399    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbr4\" (UniqueName: \"kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.436498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.436457    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.537974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.537949    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbr4\" (UniqueName: \"kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.538105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.538003    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.538105 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.538094    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.538702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.538677    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.540574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.540558    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.594056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.593994    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbr4\" (UniqueName: \"kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4\") pod \"maas-api-revoke-sub-00be7374-2qtjv\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.613684 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.613661    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:18.989109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:18.989017    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"]
Nov 01 06:47:19.352553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:19.352518    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv" event={"ID":"f97e613f-dd09-4584-b061-9b9274c4498a","Type":"ContainerStarted","Data":"505332a4b2053a44ef8a8101abf9feff30433724803122383b16a13f0411ecb3"}
Nov 01 06:47:19.352920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:19.352590    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv" event={"ID":"f97e613f-dd09-4584-b061-9b9274c4498a","Type":"ContainerStarted","Data":"25dd3b261b5b387a14348432f6b552355db71b2ff47e25b68a42b4d76de5afa8"}
Nov 01 06:47:19.468087 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:19.468034    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv" podStartSLOduration=1.468020718 podStartE2EDuration="1.468020718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:19.465077147 +0000 UTC m=+2251.032883969" watchObservedRunningTime="2026-11-01 06:47:19.468020718 +0000 UTC m=+2251.035827547"
Nov 01 06:47:20.361590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:20.361552    4468 generic.go:356] "Generic (PLEG): container finished" podID="f97e613f-dd09-4584-b061-9b9274c4498a" containerID="505332a4b2053a44ef8a8101abf9feff30433724803122383b16a13f0411ecb3" exitCode=0
Nov 01 06:47:20.362064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:20.361637    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv" event={"ID":"f97e613f-dd09-4584-b061-9b9274c4498a","Type":"ContainerDied","Data":"505332a4b2053a44ef8a8101abf9feff30433724803122383b16a13f0411ecb3"}
Nov 01 06:47:21.512798 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.512770    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:21.569876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.569842    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca\") pod \"f97e613f-dd09-4584-b061-9b9274c4498a\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") "
Nov 01 06:47:21.570071 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.570051    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4\" (UniqueName: \"kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4\") pod \"f97e613f-dd09-4584-b061-9b9274c4498a\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") "
Nov 01 06:47:21.570135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.570116    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca" pod "f97e613f-dd09-4584-b061-9b9274c4498a" (UID: "f97e613f-dd09-4584-b061-9b9274c4498a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:21.570176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.570126    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca\") pod \"f97e613f-dd09-4584-b061-9b9274c4498a\" (UID: \"f97e613f-dd09-4584-b061-9b9274c4498a\") "
Nov 01 06:47:21.570556 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.570535    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:21.572267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.572248    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4" pod "f97e613f-dd09-4584-b061-9b9274c4498a" (UID: "f97e613f-dd09-4584-b061-9b9274c4498a"). InnerVolumeSpecName "kube-api-access-ndbr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:21.572338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.572315    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca" pod "f97e613f-dd09-4584-b061-9b9274c4498a" (UID: "f97e613f-dd09-4584-b061-9b9274c4498a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:21.672283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.672203    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbr4\" (UniqueName: \"kubernetes.io/projected/f97e613f-dd09-4584-b061-9b9274c4498a-kube-api-access-ndbr4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:21.672283 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:21.672244    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97e613f-dd09-4584-b061-9b9274c4498a-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:22.375277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.375244    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv" event={"ID":"f97e613f-dd09-4584-b061-9b9274c4498a","Type":"ContainerDied","Data":"25dd3b261b5b387a14348432f6b552355db71b2ff47e25b68a42b4d76de5afa8"}
Nov 01 06:47:22.375420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.375280    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"
Nov 01 06:47:22.375420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.375321    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25dd3b261b5b387a14348432f6b552355db71b2ff47e25b68a42b4d76de5afa8"
Nov 01 06:47:22.841219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.841190    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"]
Nov 01 06:47:22.841764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.841750    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f97e613f-dd09-4584-b061-9b9274c4498a" containerName="revoke-keys"
Nov 01 06:47:22.841812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.841766    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97e613f-dd09-4584-b061-9b9274c4498a" containerName="revoke-keys"
Nov 01 06:47:22.841894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.841885    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97e613f-dd09-4584-b061-9b9274c4498a" containerName="revoke-keys"
Nov 01 06:47:22.846135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.845393    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.899187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.892798    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.899187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.892901    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.899187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.893088    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5nl\" (UniqueName: \"kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.899187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.898311    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:47:22.899187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.898398    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:47:22.914034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.914004    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:47:22.914267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.914009    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:47:22.917246 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.917217    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"]
Nov 01 06:47:22.994655 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.994623    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5nl\" (UniqueName: \"kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.994822 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.994798    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.994895 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.994880    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.995553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.995527    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:22.997338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:22.997312    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:23.050797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:23.050769    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5nl\" (UniqueName: \"kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl\") pod \"maas-api-revoke-sub-1cf0148a-sks4l\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:23.159010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:23.158936    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:23.535134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:23.535110    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"]
Nov 01 06:47:23.536638 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:23.536612    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d559858_78d8_4d3f_9034_54403599d2a9.slice/crio-a8688b9aeedae9cbaa43171b7a325b40385cf4dce90c7f94d2004df9b3d24400 WatchSource:0}: Error finding container a8688b9aeedae9cbaa43171b7a325b40385cf4dce90c7f94d2004df9b3d24400: Status 404 returned error can't find the container with id a8688b9aeedae9cbaa43171b7a325b40385cf4dce90c7f94d2004df9b3d24400
Nov 01 06:47:24.388479 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:24.388437    4468 generic.go:356] "Generic (PLEG): container finished" podID="7d559858-78d8-4d3f-9034-54403599d2a9" containerID="de5078183276e1187f48837f858e4d253bbf434019b6ab30f1fabfd5a0cc79b1" exitCode=0
Nov 01 06:47:24.388852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:24.388506    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l" event={"ID":"7d559858-78d8-4d3f-9034-54403599d2a9","Type":"ContainerDied","Data":"de5078183276e1187f48837f858e4d253bbf434019b6ab30f1fabfd5a0cc79b1"}
Nov 01 06:47:24.388852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:24.388575    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l" event={"ID":"7d559858-78d8-4d3f-9034-54403599d2a9","Type":"ContainerStarted","Data":"a8688b9aeedae9cbaa43171b7a325b40385cf4dce90c7f94d2004df9b3d24400"}
Nov 01 06:47:25.584452 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.584424    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:25.630696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.630669    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl\" (UniqueName: \"kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl\") pod \"7d559858-78d8-4d3f-9034-54403599d2a9\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") "
Nov 01 06:47:25.630800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.630780    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca\") pod \"7d559858-78d8-4d3f-9034-54403599d2a9\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") "
Nov 01 06:47:25.630854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.630839    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca\") pod \"7d559858-78d8-4d3f-9034-54403599d2a9\" (UID: \"7d559858-78d8-4d3f-9034-54403599d2a9\") "
Nov 01 06:47:25.631158 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.631138    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca" pod "7d559858-78d8-4d3f-9034-54403599d2a9" (UID: "7d559858-78d8-4d3f-9034-54403599d2a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:25.631277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.631259    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:25.632909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.632883    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca" pod "7d559858-78d8-4d3f-9034-54403599d2a9" (UID: "7d559858-78d8-4d3f-9034-54403599d2a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:25.632909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.632892    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl" pod "7d559858-78d8-4d3f-9034-54403599d2a9" (UID: "7d559858-78d8-4d3f-9034-54403599d2a9"). InnerVolumeSpecName "kube-api-access-wp5nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:25.732405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.732327    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5nl\" (UniqueName: \"kubernetes.io/projected/7d559858-78d8-4d3f-9034-54403599d2a9-kube-api-access-wp5nl\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:25.732405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:25.732357    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d559858-78d8-4d3f-9034-54403599d2a9-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:26.403057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:26.403020    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"
Nov 01 06:47:26.403217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:26.403054    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l" event={"ID":"7d559858-78d8-4d3f-9034-54403599d2a9","Type":"ContainerDied","Data":"a8688b9aeedae9cbaa43171b7a325b40385cf4dce90c7f94d2004df9b3d24400"}
Nov 01 06:47:26.403217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:26.403113    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8688b9aeedae9cbaa43171b7a325b40385cf4dce90c7f94d2004df9b3d24400"
Nov 01 06:47:31.860731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:31.860694    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"]
Nov 01 06:47:31.861123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:31.860885    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w" podUID="ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" containerName="sim" containerID="cri-o://e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b" gracePeriod=30
Nov 01 06:47:32.167984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.167947    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:47:32.318148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.318118    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls\" (UniqueName: \"kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls\") pod \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") "
Nov 01 06:47:32.318331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.318245    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv\" (UniqueName: \"kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv\") pod \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\" (UID: \"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791\") "
Nov 01 06:47:32.320935 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.320895    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls" pod "ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" (UID: "ddb5aa83-7a4c-41f5-be5a-e4c67a52b791"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:32.321151 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.320974    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv" pod "ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" (UID: "ddb5aa83-7a4c-41f5-be5a-e4c67a52b791"). InnerVolumeSpecName "kube-api-access-k5txv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:32.420391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.420311    4468 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-tls\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:32.420391 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.420339    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5txv\" (UniqueName: \"kubernetes.io/projected/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791-kube-api-access-k5txv\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:32.442737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.442706    4468 generic.go:356] "Generic (PLEG): container finished" podID="ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" containerID="e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b" exitCode=0
Nov 01 06:47:32.442855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.442762    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"
Nov 01 06:47:32.442855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.442793    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w" event={"ID":"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791","Type":"ContainerDied","Data":"e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b"}
Nov 01 06:47:32.442855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.442837    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w" event={"ID":"ddb5aa83-7a4c-41f5-be5a-e4c67a52b791","Type":"ContainerDied","Data":"086ad1c885fd23711d5d25c45ba59f695c28b26586101261a62df89262ae7e76"}
Nov 01 06:47:32.442976 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.442853    4468 scope.go:122] "RemoveContainer" containerID="e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b"
Nov 01 06:47:32.454038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.454020    4468 scope.go:122] "RemoveContainer" containerID="e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b"
Nov 01 06:47:32.454325 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:32.454306    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b\": container with ID starting with e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b not found: ID does not exist" containerID="e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b"
Nov 01 06:47:32.454434 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.454331    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b"} err="failed to get container status \"e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b\": rpc error: code = NotFound desc = could not find container \"e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b\": container with ID starting with e29304e801dba453a2263b68f2784f060344e8dec22b066c0826d577043f513b not found: ID does not exist"
Nov 01 06:47:32.605584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.605548    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"]
Nov 01 06:47:32.610993 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:32.610970    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-4e5348/e2e-usage-sim-bf7ae6-58db9f9888-l6p8w"]
Nov 01 06:47:32.947576 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:32.947541    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:47:34.425199 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425168    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"]
Nov 01 06:47:34.425762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425747    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7d559858-78d8-4d3f-9034-54403599d2a9" containerName="revoke-keys"
Nov 01 06:47:34.425809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425765    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d559858-78d8-4d3f-9034-54403599d2a9" containerName="revoke-keys"
Nov 01 06:47:34.425809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425786    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" containerName="sim"
Nov 01 06:47:34.425809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425792    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" containerName="sim"
Nov 01 06:47:34.425905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425867    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d559858-78d8-4d3f-9034-54403599d2a9" containerName="revoke-keys"
Nov 01 06:47:34.425905 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.425877    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" containerName="sim"
Nov 01 06:47:34.430387 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.430370    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.483487 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.483445    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:47:34.483698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.483542    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:47:34.487067 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.487045    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"]
Nov 01 06:47:34.532250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.532221    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:47:34.532250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.532220    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:47:34.553530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.553501    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.553672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.553656    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.553740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.553721    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8sr\" (UniqueName: \"kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.591813 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.591782    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb5aa83-7a4c-41f5-be5a-e4c67a52b791" path="/var/lib/kubelet/pods/ddb5aa83-7a4c-41f5-be5a-e4c67a52b791/volumes"
Nov 01 06:47:34.655183 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.655157    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.655343 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.655260    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.655343 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.655304    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8sr\" (UniqueName: \"kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.655953 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.655926    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.657719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.657687    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.702496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.702397    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:47:34.711514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.711484    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8sr\" (UniqueName: \"kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr\") pod \"maas-api-revoke-sub-d3cab84f-h7n6x\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:34.743561 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:34.743536    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:35.117001 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:35.116975    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"]
Nov 01 06:47:35.117734 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:35.117705    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e67296_aac8_4069_a2f1_3f8e696d5ce4.slice/crio-be4f9b80a7189b49b7ee020422cad18d1ab7e5c64c1feaf323fba446b0cb83fa WatchSource:0}: Error finding container be4f9b80a7189b49b7ee020422cad18d1ab7e5c64c1feaf323fba446b0cb83fa: Status 404 returned error can't find the container with id be4f9b80a7189b49b7ee020422cad18d1ab7e5c64c1feaf323fba446b0cb83fa
Nov 01 06:47:35.463252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:35.463220    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x" event={"ID":"72e67296-aac8-4069-a2f1-3f8e696d5ce4","Type":"ContainerStarted","Data":"be4f9b80a7189b49b7ee020422cad18d1ab7e5c64c1feaf323fba446b0cb83fa"}
Nov 01 06:47:36.471535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:36.471434    4468 generic.go:356] "Generic (PLEG): container finished" podID="72e67296-aac8-4069-a2f1-3f8e696d5ce4" containerID="0d6785b3f4444ede9395cd450c4ed35a812720698ade606d810f856260713909" exitCode=0
Nov 01 06:47:36.471535 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:36.471520    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x" event={"ID":"72e67296-aac8-4069-a2f1-3f8e696d5ce4","Type":"ContainerDied","Data":"0d6785b3f4444ede9395cd450c4ed35a812720698ade606d810f856260713909"}
Nov 01 06:47:37.573214 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.573184    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"]
Nov 01 06:47:37.579445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.579426    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.600912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.600880    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98x2\" (UniqueName: \"kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.601168 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.601147    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.601530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.601508    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.634330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.634297    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"]
Nov 01 06:47:37.691806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.691761    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:37.727900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.727871    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca\") pod \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") "
Nov 01 06:47:37.728049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.727917    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca\") pod \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") "
Nov 01 06:47:37.728117 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.728054    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr\" (UniqueName: \"kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr\") pod \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\" (UID: \"72e67296-aac8-4069-a2f1-3f8e696d5ce4\") "
Nov 01 06:47:37.728274 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.728252    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.728360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.728269    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca" pod "72e67296-aac8-4069-a2f1-3f8e696d5ce4" (UID: "72e67296-aac8-4069-a2f1-3f8e696d5ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:37.728415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.728399    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98x2\" (UniqueName: \"kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.728520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.728503    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.728709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.728660    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:37.729209 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.729187    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.730548 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.730524    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr" pod "72e67296-aac8-4069-a2f1-3f8e696d5ce4" (UID: "72e67296-aac8-4069-a2f1-3f8e696d5ce4"). InnerVolumeSpecName "kube-api-access-sg8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:37.730761 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.730744    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca" pod "72e67296-aac8-4069-a2f1-3f8e696d5ce4" (UID: "72e67296-aac8-4069-a2f1-3f8e696d5ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:37.731040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.731022    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.784771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.784748    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98x2\" (UniqueName: \"kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2\") pod \"maas-api-revoke-sub-8fbbc6d9-lp7xr\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:37.833558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.832667    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8sr\" (UniqueName: \"kubernetes.io/projected/72e67296-aac8-4069-a2f1-3f8e696d5ce4-kube-api-access-sg8sr\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:37.833558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.832701    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e67296-aac8-4069-a2f1-3f8e696d5ce4-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:37.894075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:37.894044    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:38.295403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:38.295371    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"]
Nov 01 06:47:38.296715 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:38.296691    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c874901_4cfa_41af_8be0_063c5e87f621.slice/crio-f51696c5c0a09f019ed6a5a80d75ceb6660bf4295928f3a8b502e83b8a8a1b93 WatchSource:0}: Error finding container f51696c5c0a09f019ed6a5a80d75ceb6660bf4295928f3a8b502e83b8a8a1b93: Status 404 returned error can't find the container with id f51696c5c0a09f019ed6a5a80d75ceb6660bf4295928f3a8b502e83b8a8a1b93
Nov 01 06:47:38.495090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:38.495058    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"
Nov 01 06:47:38.495257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:38.495069    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x" event={"ID":"72e67296-aac8-4069-a2f1-3f8e696d5ce4","Type":"ContainerDied","Data":"be4f9b80a7189b49b7ee020422cad18d1ab7e5c64c1feaf323fba446b0cb83fa"}
Nov 01 06:47:38.495257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:38.495175    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4f9b80a7189b49b7ee020422cad18d1ab7e5c64c1feaf323fba446b0cb83fa"
Nov 01 06:47:38.496436 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:38.496408    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr" event={"ID":"3c874901-4cfa-41af-8be0-063c5e87f621","Type":"ContainerStarted","Data":"f51696c5c0a09f019ed6a5a80d75ceb6660bf4295928f3a8b502e83b8a8a1b93"}
Nov 01 06:47:39.529605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:39.529569    4468 generic.go:356] "Generic (PLEG): container finished" podID="3c874901-4cfa-41af-8be0-063c5e87f621" containerID="74adcbd378907201360a4612f2732c6f51e0a158eaea480efe4181b0a264b0e7" exitCode=0
Nov 01 06:47:39.530070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:39.529654    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr" event={"ID":"3c874901-4cfa-41af-8be0-063c5e87f621","Type":"ContainerDied","Data":"74adcbd378907201360a4612f2732c6f51e0a158eaea480efe4181b0a264b0e7"}
Nov 01 06:47:40.692021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.691997    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:40.769051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.769022    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca\") pod \"3c874901-4cfa-41af-8be0-063c5e87f621\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") "
Nov 01 06:47:40.769227 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.769080    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca\") pod \"3c874901-4cfa-41af-8be0-063c5e87f621\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") "
Nov 01 06:47:40.769227 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.769111    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2\" (UniqueName: \"kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2\") pod \"3c874901-4cfa-41af-8be0-063c5e87f621\" (UID: \"3c874901-4cfa-41af-8be0-063c5e87f621\") "
Nov 01 06:47:40.769506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.769484    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca" pod "3c874901-4cfa-41af-8be0-063c5e87f621" (UID: "3c874901-4cfa-41af-8be0-063c5e87f621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:40.771329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.771303    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca" pod "3c874901-4cfa-41af-8be0-063c5e87f621" (UID: "3c874901-4cfa-41af-8be0-063c5e87f621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:40.771329 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.771322    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2" pod "3c874901-4cfa-41af-8be0-063c5e87f621" (UID: "3c874901-4cfa-41af-8be0-063c5e87f621"). InnerVolumeSpecName "kube-api-access-n98x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:40.870120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.870091    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:40.870120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.870117    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98x2\" (UniqueName: \"kubernetes.io/projected/3c874901-4cfa-41af-8be0-063c5e87f621-kube-api-access-n98x2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:40.870120 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:40.870127    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c874901-4cfa-41af-8be0-063c5e87f621-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:41.544876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:41.544842    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"
Nov 01 06:47:41.544876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:41.544859    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr" event={"ID":"3c874901-4cfa-41af-8be0-063c5e87f621","Type":"ContainerDied","Data":"f51696c5c0a09f019ed6a5a80d75ceb6660bf4295928f3a8b502e83b8a8a1b93"}
Nov 01 06:47:41.545097 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:41.544932    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f51696c5c0a09f019ed6a5a80d75ceb6660bf4295928f3a8b502e83b8a8a1b93"
Nov 01 06:47:45.574700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:45.574664    4468 generic.go:356] "Generic (PLEG): container finished" podID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerID="84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db" exitCode=0
Nov 01 06:47:45.575140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:45.574732    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" event={"ID":"6f86d36a-3c78-44bd-bf20-00e260f66c1e","Type":"ContainerDied","Data":"84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"}
Nov 01 06:47:45.575140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:45.575069    4468 scope.go:122] "RemoveContainer" containerID="84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"
Nov 01 06:47:45.576575 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:45.576547    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:45.576694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:45.576609    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:45.576694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:45.576666    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:47:46.596403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:46.596369    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" event={"ID":"6f86d36a-3c78-44bd-bf20-00e260f66c1e","Type":"ContainerStarted","Data":"75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a"}
Nov 01 06:47:47.176746 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:47.176719    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:47:47.177571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:47.177542    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 01 06:47:47.177694 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:47.177594    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 01 06:47:48.177270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:48.177226    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 01 06:47:48.177710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:48.177288    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 01 06:47:48.940930 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:48.940898    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:47:49.029022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.028984    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"]
Nov 01 06:47:49.029836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.029813    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3c874901-4cfa-41af-8be0-063c5e87f621" containerName="revoke-keys"
Nov 01 06:47:49.029912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.029840    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c874901-4cfa-41af-8be0-063c5e87f621" containerName="revoke-keys"
Nov 01 06:47:49.029912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.029880    4468 cpu_manager.go:414] "containerMap: removing container" podUID="72e67296-aac8-4069-a2f1-3f8e696d5ce4" containerName="revoke-keys"
Nov 01 06:47:49.029912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.029888    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e67296-aac8-4069-a2f1-3f8e696d5ce4" containerName="revoke-keys"
Nov 01 06:47:49.030049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.030036    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e67296-aac8-4069-a2f1-3f8e696d5ce4" containerName="revoke-keys"
Nov 01 06:47:49.030095 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.030063    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c874901-4cfa-41af-8be0-063c5e87f621" containerName="revoke-keys"
Nov 01 06:47:49.033633 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.033615    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.068341 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.068319    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"]
Nov 01 06:47:49.083331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.083307    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwmm\" (UniqueName: \"kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.083458 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.083334    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.083549 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.083517    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.127493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.116749    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:47:49.127493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.116799    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:47:49.135601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.135570    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:47:49.135601 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.135570    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:47:49.176696 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.176664    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 01 06:47:49.177131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.176719    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 01 06:47:49.184969 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.184942    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.185394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.185043    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwmm\" (UniqueName: \"kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.185394 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.185063    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.185701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.185680    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.187513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.187493    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.241159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.241096    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwmm\" (UniqueName: \"kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm\") pod \"maas-api-revoke-sub-ce7e13e3-5kgzh\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.348107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.348075    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:49.708490 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:49.707901    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:49.708490 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:49.707932    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:49.708490 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:49.708005    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:47:50.207985477 +0000 UTC m=+2281.775792302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:49.775155 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.775127    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"]
Nov 01 06:47:49.776036 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:49.776003    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5a471d_d3c8_4bb3_acfb_b7e008eb94b7.slice/crio-8b9ffd97696dc4cbfb6f51f49f135e56c990c5cb9249eaf8220294d1c21e2739 WatchSource:0}: Error finding container 8b9ffd97696dc4cbfb6f51f49f135e56c990c5cb9249eaf8220294d1c21e2739: Status 404 returned error can't find the container with id 8b9ffd97696dc4cbfb6f51f49f135e56c990c5cb9249eaf8220294d1c21e2739
Nov 01 06:47:49.863331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.863299    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"]
Nov 01 06:47:49.868343 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.868328    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:49.929629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.929210    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:49.929629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.929286    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:49.929629 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.929435    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km998\" (UniqueName: \"kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:49.932496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:49.932307    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"]
Nov 01 06:47:50.030594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.030503    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km998\" (UniqueName: \"kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.030758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.030632    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.030758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.030676    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.031330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.031308    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.033326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.033305    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.087108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.087075    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km998\" (UniqueName: \"kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998\") pod \"maas-api-revoke-sub-8a7f3360-fqdvl\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.177342 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.177309    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 01 06:47:50.177574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.177353    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 01 06:47:50.180567 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.180545    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:50.233941 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:50.233840    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:50.233941 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:50.233867    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:50.233941 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:50.233922    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:47:51.233904652 +0000 UTC m=+2282.801711460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:50.588205 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:47:50.588167    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab93a96_2ebd_4ce6_9442_ca3e6c78bda1.slice/crio-24ec57d64077df154441d4345d204a8f410e2d42df2d264c2be5ed6d9f409eff WatchSource:0}: Error finding container 24ec57d64077df154441d4345d204a8f410e2d42df2d264c2be5ed6d9f409eff: Status 404 returned error can't find the container with id 24ec57d64077df154441d4345d204a8f410e2d42df2d264c2be5ed6d9f409eff
Nov 01 06:47:50.591913 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.591894    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"]
Nov 01 06:47:50.677910 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.677870    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl" event={"ID":"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1","Type":"ContainerStarted","Data":"24ec57d64077df154441d4345d204a8f410e2d42df2d264c2be5ed6d9f409eff"}
Nov 01 06:47:50.679359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.679333    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh" event={"ID":"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7","Type":"ContainerStarted","Data":"cba9453e7949cb1c95778a022c4c99f830e1b4a0070cf2dda29420d68d6c84b6"}
Nov 01 06:47:50.679498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.679375    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh" event={"ID":"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7","Type":"ContainerStarted","Data":"8b9ffd97696dc4cbfb6f51f49f135e56c990c5cb9249eaf8220294d1c21e2739"}
Nov 01 06:47:50.792897 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:50.792847    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh" podStartSLOduration=2.792833457 podStartE2EDuration="2.792833457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:50.791344535 +0000 UTC m=+2282.359151386" watchObservedRunningTime="2026-11-01 06:47:50.792833457 +0000 UTC m=+2282.360640285"
Nov 01 06:47:51.177369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:51.177327    4468 patch_prober.go:29] interesting pod/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 01 06:47:51.177516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:51.177414    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 01 06:47:51.246578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:51.246545    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:51.246578 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:51.246575    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:51.246989 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:51.246629    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:47:53.246615287 +0000 UTC m=+2284.814422098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:51.687229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:51.687201    4468 generic.go:356] "Generic (PLEG): container finished" podID="5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" containerID="d3db45ef171dd42032c56ad20ef13d335c5b39a88e3a74e996daafc6c229e87e" exitCode=0
Nov 01 06:47:51.687356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:51.687277    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl" event={"ID":"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1","Type":"ContainerDied","Data":"d3db45ef171dd42032c56ad20ef13d335c5b39a88e3a74e996daafc6c229e87e"}
Nov 01 06:47:51.688820 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:51.688799    4468 generic.go:356] "Generic (PLEG): container finished" podID="8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" containerID="cba9453e7949cb1c95778a022c4c99f830e1b4a0070cf2dda29420d68d6c84b6" exitCode=0
Nov 01 06:47:51.688919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:51.688873    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh" event={"ID":"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7","Type":"ContainerDied","Data":"cba9453e7949cb1c95778a022c4c99f830e1b4a0070cf2dda29420d68d6c84b6"}
Nov 01 06:47:52.180788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.180756    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:47:52.181775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.181749    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 06:47:52.861616 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.861592    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:52.864920 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.864901    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:52.968383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968325    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca\") pod \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") "
Nov 01 06:47:52.968383 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968358    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca\") pod \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") "
Nov 01 06:47:52.968600 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968521    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm\" (UniqueName: \"kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm\") pod \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") "
Nov 01 06:47:52.968661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968625    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998\" (UniqueName: \"kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998\") pod \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") "
Nov 01 06:47:52.968727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968708    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca" pod "8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" (UID: "8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:52.968785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968733    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca\") pod \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\" (UID: \"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7\") "
Nov 01 06:47:52.968785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968740    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca" pod "5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" (UID: "5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:47:52.968785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.968766    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca\") pod \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\" (UID: \"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1\") "
Nov 01 06:47:52.969288 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.969263    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.969456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.969292    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.970869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.970847    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca" pod "8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" (UID: "8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:52.970970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.970924    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm" pod "8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" (UID: "8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7"). InnerVolumeSpecName "kube-api-access-pbwmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:52.970970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.970947    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998" pod "5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" (UID: "5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1"). InnerVolumeSpecName "kube-api-access-km998". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:52.970970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:52.970957    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca" pod "5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" (UID: "5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:53.070872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.070841    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbwmm\" (UniqueName: \"kubernetes.io/projected/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-kube-api-access-pbwmm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:53.070872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.070868    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km998\" (UniqueName: \"kubernetes.io/projected/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-kube-api-access-km998\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:53.071040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.070879    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:53.071040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.070889    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:53.273688 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:53.273593    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:53.273688 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:53.273630    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:53.273869 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:53.273695    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:47:57.27367628 +0000 UTC m=+2288.841483103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:53.703689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.703656    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"
Nov 01 06:47:53.703856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.703687    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh" event={"ID":"8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7","Type":"ContainerDied","Data":"8b9ffd97696dc4cbfb6f51f49f135e56c990c5cb9249eaf8220294d1c21e2739"}
Nov 01 06:47:53.703856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.703733    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b9ffd97696dc4cbfb6f51f49f135e56c990c5cb9249eaf8220294d1c21e2739"
Nov 01 06:47:53.705146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.705121    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl" event={"ID":"5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1","Type":"ContainerDied","Data":"24ec57d64077df154441d4345d204a8f410e2d42df2d264c2be5ed6d9f409eff"}
Nov 01 06:47:53.705257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.705147    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"
Nov 01 06:47:53.705257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:53.705160    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ec57d64077df154441d4345d204a8f410e2d42df2d264c2be5ed6d9f409eff"
Nov 01 06:47:56.261128 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:47:56.261095    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:47:57.326299 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:57.326269    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:57.326299 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:57.326301    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:47:57.326736 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:47:57.326355    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:48:05.326340669 +0000 UTC m=+2296.894147488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : secret "agc-cred-1a1f3593a873" not found
Nov 01 06:48:02.814524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.814407    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"]
Nov 01 06:48:02.815525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.815298    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" containerName="revoke-keys"
Nov 01 06:48:02.815525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.815323    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" containerName="revoke-keys"
Nov 01 06:48:02.815525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.815342    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" containerName="revoke-keys"
Nov 01 06:48:02.815525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.815349    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" containerName="revoke-keys"
Nov 01 06:48:02.815525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.815493    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" containerName="revoke-keys"
Nov 01 06:48:02.815525 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.815506    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" containerName="revoke-keys"
Nov 01 06:48:02.822747 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.822717    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"]
Nov 01 06:48:02.822907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.822843    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.894649 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.894616    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.894837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.894712    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.894886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.894867    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9k6\" (UniqueName: \"kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.924540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.924501    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:48:02.924540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.924528    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:48:02.924540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.924507    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:48:02.924793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.924640    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:48:02.949284 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:02.949248    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:48:02.995660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.995624    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9k6\" (UniqueName: \"kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.995836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.995732    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.995836 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.995794    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.996524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.996460    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:02.998339 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:02.998320    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:03.071544 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.071423    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9k6\" (UniqueName: \"kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6\") pod \"maas-api-revoke-sub-72c8871b-qmtdx\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:03.141862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.141829    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:03.337768 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.337696    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"]
Nov 01 06:48:03.342846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.342827    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.398221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.398192    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"]
Nov 01 06:48:03.504789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.504756    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.505034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.505013    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.505080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.505045    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5c49\" (UniqueName: \"kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.568494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.568449    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"]
Nov 01 06:48:03.569130 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:48:03.569100    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe71916_e0c6_4b6c_99d1_0449cb9b2a13.slice/crio-fa2f91fe960c06adfeef6f35eac72b20b509cdd1f6e9c13aa184ffd9a32772cd WatchSource:0}: Error finding container fa2f91fe960c06adfeef6f35eac72b20b509cdd1f6e9c13aa184ffd9a32772cd: Status 404 returned error can't find the container with id fa2f91fe960c06adfeef6f35eac72b20b509cdd1f6e9c13aa184ffd9a32772cd
Nov 01 06:48:03.605872 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.605843    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.606021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.605877    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5c49\" (UniqueName: \"kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.606021 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.605918    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.606606 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.606586    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.608295 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.608275    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.662377 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.662343    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5c49\" (UniqueName: \"kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49\") pod \"maas-api-revoke-sub-b0526811-mnz4b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:03.808237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.808204    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx" event={"ID":"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13","Type":"ContainerStarted","Data":"fa2f91fe960c06adfeef6f35eac72b20b509cdd1f6e9c13aa184ffd9a32772cd"}
Nov 01 06:48:03.956287 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:03.956207    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:04.326871 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:04.326843    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"]
Nov 01 06:48:04.327612 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:48:04.327589    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399fe1f3_0227_458b_9c08_c8c63e4ce67b.slice/crio-38c5a4f025981113df6c5e3ac7f08c7609a5f8f418de8a5c0aa46f1ee74ced69 WatchSource:0}: Error finding container 38c5a4f025981113df6c5e3ac7f08c7609a5f8f418de8a5c0aa46f1ee74ced69: Status 404 returned error can't find the container with id 38c5a4f025981113df6c5e3ac7f08c7609a5f8f418de8a5c0aa46f1ee74ced69
Nov 01 06:48:04.815850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:04.815819    4468 generic.go:356] "Generic (PLEG): container finished" podID="cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" containerID="ac8740368269f237764565253df3790c0020dea32523b9743a9928cc6a210344" exitCode=0
Nov 01 06:48:04.816029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:04.815893    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx" event={"ID":"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13","Type":"ContainerDied","Data":"ac8740368269f237764565253df3790c0020dea32523b9743a9928cc6a210344"}
Nov 01 06:48:04.817512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:04.817485    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b" event={"ID":"399fe1f3-0227-458b-9c08-c8c63e4ce67b","Type":"ContainerStarted","Data":"a76edff6d9f8ab22015e7c19d19f45cbb5ab5c1c6110efbdac2c5f2771a2be98"}
Nov 01 06:48:04.817615 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:04.817529    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b" event={"ID":"399fe1f3-0227-458b-9c08-c8c63e4ce67b","Type":"ContainerStarted","Data":"38c5a4f025981113df6c5e3ac7f08c7609a5f8f418de8a5c0aa46f1ee74ced69"}
Nov 01 06:48:05.054806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:05.054754    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b" podStartSLOduration=2.054738285 podStartE2EDuration="2.054738285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:05.052416437 +0000 UTC m=+2296.620223275" watchObservedRunningTime="2026-11-01 06:48:05.054738285 +0000 UTC m=+2296.622545113"
Nov 01 06:48:05.343517 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:05.343428    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:48:05.343517 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:05.343456    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-2b38f786e9be: secret "agc-cred-2b38f786e9be" not found
Nov 01 06:48:05.343517 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:05.343486    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: [secret "agc-cred-1a1f3593a873" not found, secret "agc-cred-2b38f786e9be" not found]
Nov 01 06:48:05.343813 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:05.343545    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:48:21.343531 +0000 UTC m=+2312.911337808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : [secret "agc-cred-1a1f3593a873" not found, secret "agc-cred-2b38f786e9be" not found]
Nov 01 06:48:05.827917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:05.827886    4468 generic.go:356] "Generic (PLEG): container finished" podID="399fe1f3-0227-458b-9c08-c8c63e4ce67b" containerID="a76edff6d9f8ab22015e7c19d19f45cbb5ab5c1c6110efbdac2c5f2771a2be98" exitCode=0
Nov 01 06:48:05.828116 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:05.827972    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b" event={"ID":"399fe1f3-0227-458b-9c08-c8c63e4ce67b","Type":"ContainerDied","Data":"a76edff6d9f8ab22015e7c19d19f45cbb5ab5c1c6110efbdac2c5f2771a2be98"}
Nov 01 06:48:05.968992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:05.968967    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:06.051856 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.051828    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca\") pod \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") "
Nov 01 06:48:06.052040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.051924    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca\") pod \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") "
Nov 01 06:48:06.052040 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.051971    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6\" (UniqueName: \"kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6\") pod \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\" (UID: \"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13\") "
Nov 01 06:48:06.052245 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.052221    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca" pod "cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" (UID: "cbe71916-e0c6-4b6c-99d1-0449cb9b2a13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:06.052435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.052416    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:06.054075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.054040    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca" pod "cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" (UID: "cbe71916-e0c6-4b6c-99d1-0449cb9b2a13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:06.054190 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.054123    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6" pod "cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" (UID: "cbe71916-e0c6-4b6c-99d1-0449cb9b2a13"). InnerVolumeSpecName "kube-api-access-wf9k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:06.157550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.156526    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:06.157550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.156568    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf9k6\" (UniqueName: \"kubernetes.io/projected/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13-kube-api-access-wf9k6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:06.836182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.836147    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"
Nov 01 06:48:06.836182 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.836174    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx" event={"ID":"cbe71916-e0c6-4b6c-99d1-0449cb9b2a13","Type":"ContainerDied","Data":"fa2f91fe960c06adfeef6f35eac72b20b509cdd1f6e9c13aa184ffd9a32772cd"}
Nov 01 06:48:06.836405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.836233    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2f91fe960c06adfeef6f35eac72b20b509cdd1f6e9c13aa184ffd9a32772cd"
Nov 01 06:48:06.983440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:06.983413    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:07.066792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.066765    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca\") pod \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") "
Nov 01 06:48:07.066923 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.066904    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49\" (UniqueName: \"kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49\") pod \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") "
Nov 01 06:48:07.066967 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.066934    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca\") pod \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\" (UID: \"399fe1f3-0227-458b-9c08-c8c63e4ce67b\") "
Nov 01 06:48:07.067325 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.067301    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca" pod "399fe1f3-0227-458b-9c08-c8c63e4ce67b" (UID: "399fe1f3-0227-458b-9c08-c8c63e4ce67b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:07.069032 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.069014    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49" pod "399fe1f3-0227-458b-9c08-c8c63e4ce67b" (UID: "399fe1f3-0227-458b-9c08-c8c63e4ce67b"). InnerVolumeSpecName "kube-api-access-f5c49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:07.069128 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.069051    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca" pod "399fe1f3-0227-458b-9c08-c8c63e4ce67b" (UID: "399fe1f3-0227-458b-9c08-c8c63e4ce67b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:07.168456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.168375    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:07.168456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.168406    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5c49\" (UniqueName: \"kubernetes.io/projected/399fe1f3-0227-458b-9c08-c8c63e4ce67b-kube-api-access-f5c49\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:07.168456 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.168415    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fe1f3-0227-458b-9c08-c8c63e4ce67b-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:07.843704 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.843671    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b" event={"ID":"399fe1f3-0227-458b-9c08-c8c63e4ce67b","Type":"ContainerDied","Data":"38c5a4f025981113df6c5e3ac7f08c7609a5f8f418de8a5c0aa46f1ee74ced69"}
Nov 01 06:48:07.843919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.843728    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38c5a4f025981113df6c5e3ac7f08c7609a5f8f418de8a5c0aa46f1ee74ced69"
Nov 01 06:48:07.843919 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:07.843677    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"
Nov 01 06:48:14.501827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.501790    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"]
Nov 01 06:48:14.502359 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.502344    4468 cpu_manager.go:414] "containerMap: removing container" podUID="399fe1f3-0227-458b-9c08-c8c63e4ce67b" containerName="revoke-keys"
Nov 01 06:48:14.502403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.502361    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399fe1f3-0227-458b-9c08-c8c63e4ce67b" containerName="revoke-keys"
Nov 01 06:48:14.502403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.502390    4468 cpu_manager.go:414] "containerMap: removing container" podUID="cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" containerName="revoke-keys"
Nov 01 06:48:14.502403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.502395    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" containerName="revoke-keys"
Nov 01 06:48:14.502550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.502491    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" containerName="revoke-keys"
Nov 01 06:48:14.502550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.502504    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="399fe1f3-0227-458b-9c08-c8c63e4ce67b" containerName="revoke-keys"
Nov 01 06:48:14.505627 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.505612    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.558613 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.558583    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wjbw\" (UniqueName: \"kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.558805 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.558789    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.558916 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.558901    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.562777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.562750    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"]
Nov 01 06:48:14.607700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.607665    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:48:14.607700 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.607675    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:48:14.607939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.607718    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:48:14.607939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.607738    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:48:14.659851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.659811    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.660053 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.659952    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.660167 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.660119    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wjbw\" (UniqueName: \"kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.660603 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.660576    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.662677 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.662657    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.716384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.716353    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wjbw\" (UniqueName: \"kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw\") pod \"maas-api-revoke-sub-b7bb4d35-t589v\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:14.819036 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:14.819002    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:15.196671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:15.196641    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"]
Nov 01 06:48:15.197212 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:48:15.197181    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc638b09b_ab4e_4266_9be6_fdb3d2f09f74.slice/crio-9449863a0a663c0dfd1f896cc222946df55bb4c667f4ceb2085ee04fc7fe85ef WatchSource:0}: Error finding container 9449863a0a663c0dfd1f896cc222946df55bb4c667f4ceb2085ee04fc7fe85ef: Status 404 returned error can't find the container with id 9449863a0a663c0dfd1f896cc222946df55bb4c667f4ceb2085ee04fc7fe85ef
Nov 01 06:48:15.904447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:15.904409    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v" event={"ID":"c638b09b-ab4e-4266-9be6-fdb3d2f09f74","Type":"ContainerStarted","Data":"d955ca28d1e0d9312f0085ef248e275ecc93f68dec9472b405a6ed209ab08597"}
Nov 01 06:48:15.905086 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:15.905055    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v" event={"ID":"c638b09b-ab4e-4266-9be6-fdb3d2f09f74","Type":"ContainerStarted","Data":"9449863a0a663c0dfd1f896cc222946df55bb4c667f4ceb2085ee04fc7fe85ef"}
Nov 01 06:48:16.027950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.027886    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v" podStartSLOduration=2.027867132 podStartE2EDuration="2.027867132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:16.021754353 +0000 UTC m=+2307.589561184" watchObservedRunningTime="2026-11-01 06:48:16.027867132 +0000 UTC m=+2307.595673962"
Nov 01 06:48:16.231481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.231441    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"]
Nov 01 06:48:16.235384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.235369    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.293877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.292131    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"]
Nov 01 06:48:16.297818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.297791    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.298052 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.298025    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.298170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.298152    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvm2\" (UniqueName: \"kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.399059 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.399023    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.399248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.399137    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.399248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.399189    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvm2\" (UniqueName: \"kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.399791 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.399767    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.401479 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.401450    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.455275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.455211    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvm2\" (UniqueName: \"kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2\") pod \"maas-api-revoke-sub-4f7747dd-v5bz8\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.548715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.548681    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:16.917489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.914044    4468 generic.go:356] "Generic (PLEG): container finished" podID="c638b09b-ab4e-4266-9be6-fdb3d2f09f74" containerID="d955ca28d1e0d9312f0085ef248e275ecc93f68dec9472b405a6ed209ab08597" exitCode=0
Nov 01 06:48:16.917489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.914294    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v" event={"ID":"c638b09b-ab4e-4266-9be6-fdb3d2f09f74","Type":"ContainerDied","Data":"d955ca28d1e0d9312f0085ef248e275ecc93f68dec9472b405a6ed209ab08597"}
Nov 01 06:48:16.931689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:16.931663    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"]
Nov 01 06:48:16.935210 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:48:16.935160    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e008f13_8f3e_4e9f_9f31_1350c1a6a340.slice/crio-8206e295b3632018a0b112d33ef99f7750009c54bf69fe8693f20317ca042c5c WatchSource:0}: Error finding container 8206e295b3632018a0b112d33ef99f7750009c54bf69fe8693f20317ca042c5c: Status 404 returned error can't find the container with id 8206e295b3632018a0b112d33ef99f7750009c54bf69fe8693f20317ca042c5c
Nov 01 06:48:17.925883 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:17.925846    4468 generic.go:356] "Generic (PLEG): container finished" podID="7e008f13-8f3e-4e9f-9f31-1350c1a6a340" containerID="cf15d35fae6c463ae6396cc790b7d901f71eafae77c728ef6b1c02456a0c47fe" exitCode=0
Nov 01 06:48:17.926309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:17.925931    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8" event={"ID":"7e008f13-8f3e-4e9f-9f31-1350c1a6a340","Type":"ContainerDied","Data":"cf15d35fae6c463ae6396cc790b7d901f71eafae77c728ef6b1c02456a0c47fe"}
Nov 01 06:48:17.926309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:17.925978    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8" event={"ID":"7e008f13-8f3e-4e9f-9f31-1350c1a6a340","Type":"ContainerStarted","Data":"8206e295b3632018a0b112d33ef99f7750009c54bf69fe8693f20317ca042c5c"}
Nov 01 06:48:18.080512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.080487    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:18.120263 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.120239    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca\") pod \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") "
Nov 01 06:48:18.120403 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.120298    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca\") pod \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") "
Nov 01 06:48:18.120512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.120493    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw\" (UniqueName: \"kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw\") pod \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\" (UID: \"c638b09b-ab4e-4266-9be6-fdb3d2f09f74\") "
Nov 01 06:48:18.120642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.120623    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca" pod "c638b09b-ab4e-4266-9be6-fdb3d2f09f74" (UID: "c638b09b-ab4e-4266-9be6-fdb3d2f09f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:18.120958 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.120941    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:18.122435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.122417    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca" pod "c638b09b-ab4e-4266-9be6-fdb3d2f09f74" (UID: "c638b09b-ab4e-4266-9be6-fdb3d2f09f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:18.122579 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.122558    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw" pod "c638b09b-ab4e-4266-9be6-fdb3d2f09f74" (UID: "c638b09b-ab4e-4266-9be6-fdb3d2f09f74"). InnerVolumeSpecName "kube-api-access-4wjbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:18.222514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.222482    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wjbw\" (UniqueName: \"kubernetes.io/projected/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-kube-api-access-4wjbw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:18.222514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.222515    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c638b09b-ab4e-4266-9be6-fdb3d2f09f74-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:18.933435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.933361    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"
Nov 01 06:48:18.933435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.933399    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v" event={"ID":"c638b09b-ab4e-4266-9be6-fdb3d2f09f74","Type":"ContainerDied","Data":"9449863a0a663c0dfd1f896cc222946df55bb4c667f4ceb2085ee04fc7fe85ef"}
Nov 01 06:48:18.933435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:18.933439    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9449863a0a663c0dfd1f896cc222946df55bb4c667f4ceb2085ee04fc7fe85ef"
Nov 01 06:48:19.148546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.148520    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:19.246084 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.245996    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2\" (UniqueName: \"kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2\") pod \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") "
Nov 01 06:48:19.246245 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.246132    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca\") pod \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") "
Nov 01 06:48:19.246245 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.246161    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca\") pod \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\" (UID: \"7e008f13-8f3e-4e9f-9f31-1350c1a6a340\") "
Nov 01 06:48:19.246506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.246450    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca" pod "7e008f13-8f3e-4e9f-9f31-1350c1a6a340" (UID: "7e008f13-8f3e-4e9f-9f31-1350c1a6a340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:19.246691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.246595    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:19.248276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.248252    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca" pod "7e008f13-8f3e-4e9f-9f31-1350c1a6a340" (UID: "7e008f13-8f3e-4e9f-9f31-1350c1a6a340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:19.248385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.248362    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2" pod "7e008f13-8f3e-4e9f-9f31-1350c1a6a340" (UID: "7e008f13-8f3e-4e9f-9f31-1350c1a6a340"). InnerVolumeSpecName "kube-api-access-7zvm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:19.348115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.348085    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvm2\" (UniqueName: \"kubernetes.io/projected/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-kube-api-access-7zvm2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:19.348115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.348112    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e008f13-8f3e-4e9f-9f31-1350c1a6a340-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:19.940390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.940361    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"
Nov 01 06:48:19.940800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.940404    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8" event={"ID":"7e008f13-8f3e-4e9f-9f31-1350c1a6a340","Type":"ContainerDied","Data":"8206e295b3632018a0b112d33ef99f7750009c54bf69fe8693f20317ca042c5c"}
Nov 01 06:48:19.940800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:19.940454    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8206e295b3632018a0b112d33ef99f7750009c54bf69fe8693f20317ca042c5c"
Nov 01 06:48:21.378244 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:21.378212    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-120370fe39d3: secret "agc-cred-120370fe39d3" not found
Nov 01 06:48:21.378244 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:21.378240    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-1a1f3593a873: secret "agc-cred-1a1f3593a873" not found
Nov 01 06:48:21.378244 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:21.378247    4468 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-dc6e48/agc-cred-2b38f786e9be: secret "agc-cred-2b38f786e9be" not found
Nov 01 06:48:21.378738 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:21.378260    4468 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl: [secret "agc-cred-120370fe39d3" not found, secret "agc-cred-1a1f3593a873" not found, secret "agc-cred-2b38f786e9be" not found]
Nov 01 06:48:21.378738 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:21.378311    4468 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials podName:9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48 nodeName:}" failed. No retries permitted until 2026-11-01 06:48:53.378298462 +0000 UTC m=+2344.946105282 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48") : [secret "agc-cred-120370fe39d3" not found, secret "agc-cred-1a1f3593a873" not found, secret "agc-cred-2b38f786e9be" not found]
Nov 01 06:48:32.766217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:32.766185    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"]
Nov 01 06:48:32.946920 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:32.946871    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:48:37.766679 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:37.766640    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" podUID="9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" containerName="payload-processing" containerID="cri-o://bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5" gracePeriod=25
Nov 01 06:48:38.008144 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.008121    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:48:38.084019 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.083992    4468 generic.go:356] "Generic (PLEG): container finished" podID="9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" containerID="bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5" exitCode=0
Nov 01 06:48:38.084160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.084055    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"
Nov 01 06:48:38.084160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.084077    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" event={"ID":"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48","Type":"ContainerDied","Data":"bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5"}
Nov 01 06:48:38.084160 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.084147    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl" event={"ID":"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48","Type":"ContainerDied","Data":"70eb5ed4a589ffb0a4091352e4a00ac6854dda2e7dd8c6f8c826da57acaa0639"}
Nov 01 06:48:38.084279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.084168    4468 scope.go:122] "RemoveContainer" containerID="bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5"
Nov 01 06:48:38.093319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.093301    4468 scope.go:122] "RemoveContainer" containerID="bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5"
Nov 01 06:48:38.093605 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:48:38.093585    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5\": container with ID starting with bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5 not found: ID does not exist" containerID="bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5"
Nov 01 06:48:38.093652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.093616    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5"} err="failed to get container status \"bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5\": rpc error: code = NotFound desc = could not find container \"bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5\": container with ID starting with bf97b336543502ccdb0af9616b4e5d7f505428992790256cae957fdfce82a2a5 not found: ID does not exist"
Nov 01 06:48:38.105918 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.105899    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7\") pod \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") "
Nov 01 06:48:38.106008 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.105998    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume\") pod \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") "
Nov 01 06:48:38.106073 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.106014    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume\") pod \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") "
Nov 01 06:48:38.106134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.106084    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials\") pod \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\" (UID: \"9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48\") "
Nov 01 06:48:38.106374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.106350    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume" pod "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:38.106459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.106438    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume" pod "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:38.108133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.108112    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7" pod "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48"). InnerVolumeSpecName "kube-api-access-pwtg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:38.108327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.108309    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials" pod "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" (UID: "9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:38.207252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.207223    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtg7\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-kube-api-access-pwtg7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:38.207252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.207249    4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-routing-overlay-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:38.207414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.207259    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:38.207414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.207268    4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48-provider-credentials\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:38.505138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.505106    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"]
Nov 01 06:48:38.554708 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.554683    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-79c8hxgnl"]
Nov 01 06:48:38.590054 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:38.590028    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" path="/var/lib/kubelet/pods/9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48/volumes"
Nov 01 06:48:40.680950 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.680919    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.898767    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"]
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899344    4468 cpu_manager.go:414] "containerMap: removing container" podUID="9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" containerName="payload-processing"
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899359    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" containerName="payload-processing"
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899404    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7e008f13-8f3e-4e9f-9f31-1350c1a6a340" containerName="revoke-keys"
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899411    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e008f13-8f3e-4e9f-9f31-1350c1a6a340" containerName="revoke-keys"
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899424    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c638b09b-ab4e-4266-9be6-fdb3d2f09f74" containerName="revoke-keys"
Nov 01 06:48:40.899491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899432    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c638b09b-ab4e-4266-9be6-fdb3d2f09f74" containerName="revoke-keys"
Nov 01 06:48:40.899956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899559    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e008f13-8f3e-4e9f-9f31-1350c1a6a340" containerName="revoke-keys"
Nov 01 06:48:40.899956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899575    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c638b09b-ab4e-4266-9be6-fdb3d2f09f74" containerName="revoke-keys"
Nov 01 06:48:40.899956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.899582    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7f04ad-5fda-4ed8-81d3-f8cc1235fd48" containerName="payload-processing"
Nov 01 06:48:40.904512 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.904481    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:40.975300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:40.975229    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-581487-openshift-default-dockercfg-wxzmj\""
Nov 01 06:48:41.012650 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.012621    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"]
Nov 01 06:48:41.052085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052045    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052094    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcb5z\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052175    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052244 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052208    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052306    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052342    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052415 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052363    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052498    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.052533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.052529    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.153758 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153731    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.153925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153776    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcb5z\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.153925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153819    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.153925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153847    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.153925 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153909    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153945    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.153973    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154071    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154148 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154113    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154165    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154272    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154523    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154662    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.154865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.154846    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.156296 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.156278    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.156536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.156519    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.211177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.211143    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcb5z\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.211334 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.211316    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token\") pod \"e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") " pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.224319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.224295    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:41.687816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.684902    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"]
Nov 01 06:48:41.696543 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:48:41.690479    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4147bbc5_e12f_45e2_8f0e_87f046fc8bdc.slice/crio-aea54d618ccc54c2cbf5b5e9eb489fa12115597ef0d795925d41f84a70e86da2 WatchSource:0}: Error finding container aea54d618ccc54c2cbf5b5e9eb489fa12115597ef0d795925d41f84a70e86da2: Status 404 returned error can't find the container with id aea54d618ccc54c2cbf5b5e9eb489fa12115597ef0d795925d41f84a70e86da2
Nov 01 06:48:41.704557 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.704502    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:48:41.704691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.704604    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:48:41.704691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:41.704648    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:48:42.113361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:42.113323    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" event={"ID":"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc","Type":"ContainerStarted","Data":"007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"}
Nov 01 06:48:42.113572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:42.113381    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" event={"ID":"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc","Type":"ContainerStarted","Data":"aea54d618ccc54c2cbf5b5e9eb489fa12115597ef0d795925d41f84a70e86da2"}
Nov 01 06:48:42.224880 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:42.224847    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:42.229929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:42.229908    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:42.239234 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:42.239191    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podStartSLOduration=2.239174629 podStartE2EDuration="2.239174629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:42.234982463 +0000 UTC m=+2333.802789304" watchObservedRunningTime="2026-11-01 06:48:42.239174629 +0000 UTC m=+2333.806981488"
Nov 01 06:48:43.120617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:43.120588    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:43.121552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:43.121534    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:48:51.152507 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:51.152444    4468 scope.go:122] "RemoveContainer" containerID="889dd85e2af73334920da8c73c909e2e018ab3ff41a1120f76f27ea7f385d551"
Nov 01 06:48:51.167789 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:51.167760    4468 scope.go:122] "RemoveContainer" containerID="0a41fd9ad12b9c548889fdd81c517e141321294a7be5dba4bfae448f9724bb9e"
Nov 01 06:48:56.096493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:56.085933    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"]
Nov 01 06:48:56.156740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:56.156704    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93bfa728-5xfvc"]
Nov 01 06:48:56.616492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:48:56.614298    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f55a17-d5f1-44f4-b24b-ebb04fff4f13" path="/var/lib/kubelet/pods/e1f55a17-d5f1-44f4-b24b-ebb04fff4f13/volumes"
Nov 01 06:49:02.948148 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:49:02.948107    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:49:06.047132 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:06.047096    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:49:15.045550 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:15.045517    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"]
Nov 01 06:49:15.094080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:15.094048    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1d3503-cv2cs"]
Nov 01 06:49:16.589542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:16.589507    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedfecb8-266c-4835-be8f-d4a39013cd39" path="/var/lib/kubelet/pods/bedfecb8-266c-4835-be8f-d4a39013cd39/volumes"
Nov 01 06:49:26.045992 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:26.045958    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"]
Nov 01 06:49:26.095078 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:26.095050    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbc38c69-hgtfn"]
Nov 01 06:49:26.596399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:26.596359    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceeedf05-bc6e-4086-8672-98cf79d6a139" path="/var/lib/kubelet/pods/ceeedf05-bc6e-4086-8672-98cf79d6a139/volumes"
Nov 01 06:49:27.522257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:27.522163    4468 generic.go:356] "Generic (PLEG): container finished" podID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerID="857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466" exitCode=0
Nov 01 06:49:27.522257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:27.522236    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" event={"ID":"8161e4f0-456c-427f-ae79-b2fa97b5ea6d","Type":"ContainerDied","Data":"857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"}
Nov 01 06:49:27.522873 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:27.522756    4468 scope.go:122] "RemoveContainer" containerID="857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"
Nov 01 06:49:27.524539 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:27.524510    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:49:27.524617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:27.524589    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:49:27.524657 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:27.524622    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:49:28.534127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:28.534093    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" event={"ID":"8161e4f0-456c-427f-ae79-b2fa97b5ea6d","Type":"ContainerStarted","Data":"18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203"}
Nov 01 06:49:28.853327 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:28.853298    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:49:28.855229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:28.855205    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 06:49:28.855398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:28.855373    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 06:49:29.858490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:29.857854    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 06:49:29.858490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:29.857920    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 06:49:30.861887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:30.861849    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 06:49:30.862534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:30.862494    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 06:49:31.864491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:31.864217    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 06:49:31.864491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:31.864291    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 06:49:32.215545 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:32.215036    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:49:32.854010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:32.853972    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 06:49:32.854191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:32.854029    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 06:49:32.954235 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:49:32.954197    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:49:33.854016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:33.853984    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 06:49:33.854232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:33.854036    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 06:49:34.857857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:34.857827    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:49:34.858934 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:49:34.858909    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:50:02.954444 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:50:02.954409    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:50:03.130929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:03.130887    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"]
Nov 01 06:50:03.131186 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:03.131151    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy" containerID="cri-o://777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49" gracePeriod=30
Nov 01 06:50:08.404863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.404829    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:50:08.550928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.550895    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.550966    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551051    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551090    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551116    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551142    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551172    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551198    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:08.551398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551312    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551342    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo\") pod \"4d705c95-3058-430e-8915-ef05b0bb6175\" (UID: \"4d705c95-3058-430e-8915-ef05b0bb6175\") "
Nov 01 06:50:08.551398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551362    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:08.551652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551373    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:08.551652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551448    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:08.551763 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551647    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:08.551816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551798    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.551861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551820    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.551861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551835    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.551861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551848    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d705c95-3058-430e-8915-ef05b0bb6175-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.551976 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.551860    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.553688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.553666    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:08.553688 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.553679    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:50:08.554118 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.554094    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "kube-api-access-nxswb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:08.554208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.554130    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token" pod "4d705c95-3058-430e-8915-ef05b0bb6175" (UID: "4d705c95-3058-430e-8915-ef05b0bb6175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:08.653555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.653490    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxswb\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-kube-api-access-nxswb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.653555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.653525    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d705c95-3058-430e-8915-ef05b0bb6175-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.653555 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.653541    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d705c95-3058-430e-8915-ef05b0bb6175-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.653752 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.653555    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d705c95-3058-430e-8915-ef05b0bb6175-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:08.929863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.929773    4468 generic.go:356] "Generic (PLEG): container finished" podID="4d705c95-3058-430e-8915-ef05b0bb6175" containerID="777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49" exitCode=0
Nov 01 06:50:08.929863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.929820    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"
Nov 01 06:50:08.930088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.929862    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" event={"ID":"4d705c95-3058-430e-8915-ef05b0bb6175","Type":"ContainerDied","Data":"777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49"}
Nov 01 06:50:08.930088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.929922    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6" event={"ID":"4d705c95-3058-430e-8915-ef05b0bb6175","Type":"ContainerDied","Data":"f9d48311cd1e99f1ba539dad87728cd09fc67ab41ec0bef4cac226a002cfa9b9"}
Nov 01 06:50:08.930088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.929947    4468 scope.go:122] "RemoveContainer" containerID="777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49"
Nov 01 06:50:08.941964 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.941933    4468 scope.go:122] "RemoveContainer" containerID="37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"
Nov 01 06:50:08.950849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.950827    4468 scope.go:122] "RemoveContainer" containerID="777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49"
Nov 01 06:50:08.951166 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:50:08.951147    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49\": container with ID starting with 777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49 not found: ID does not exist" containerID="777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49"
Nov 01 06:50:08.951232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.951178    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49"} err="failed to get container status \"777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49\": rpc error: code = NotFound desc = could not find container \"777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49\": container with ID starting with 777f0f2061f378fcf2d2214f7aa310fd674d5ffce9e60bc79c98211f1ca23d49 not found: ID does not exist"
Nov 01 06:50:08.951232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.951196    4468 scope.go:122] "RemoveContainer" containerID="37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"
Nov 01 06:50:08.951445 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:50:08.951423    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165\": container with ID starting with 37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165 not found: ID does not exist" containerID="37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"
Nov 01 06:50:08.951522 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:08.951454    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165"} err="failed to get container status \"37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165\": rpc error: code = NotFound desc = could not find container \"37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165\": container with ID starting with 37222445e15aaa45a2060c4b3bed1ea2a00922cc4fdc4f8dfb7177e73dfc8165 not found: ID does not exist"
Nov 01 06:50:09.095476 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:09.095426    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"]
Nov 01 06:50:09.144971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:09.144937    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-47bc8f48-openshift-default-7767fd945c-q2cj6"]
Nov 01 06:50:10.591686 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:10.591658    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" path="/var/lib/kubelet/pods/4d705c95-3058-430e-8915-ef05b0bb6175/volumes"
Nov 01 06:50:11.959495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:11.959442    4468 generic.go:356] "Generic (PLEG): container finished" podID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerID="007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446" exitCode=0
Nov 01 06:50:11.960026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:11.959526    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" event={"ID":"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc","Type":"ContainerDied","Data":"007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"}
Nov 01 06:50:11.960026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:11.959972    4468 scope.go:122] "RemoveContainer" containerID="007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"
Nov 01 06:50:11.961355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:11.961326    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:50:11.961414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:11.961394    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:50:11.961492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:11.961425    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:50:12.968114 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:12.968082    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" event={"ID":"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc","Type":"ContainerStarted","Data":"384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288"}
Nov 01 06:50:13.122621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:13.122584    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:50:13.227867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:13.227777    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:50:13.228809 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:13.228783    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:50:13.228947 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:13.228833    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:50:14.224811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:14.224777    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:50:14.227594 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:14.227552    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:50:15.227854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:15.227820    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:50:15.228333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:15.227892    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:50:16.227901 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:16.227863    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:50:16.228542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:16.228509    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:50:16.997664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:16.997633    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"]
Nov 01 06:50:16.997848 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:16.997827    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy" containerID="cri-o://858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b" gracePeriod=30
Nov 01 06:50:17.227826 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:17.227790    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:50:17.228115 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:17.228087    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:50:18.229326 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:18.229299    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:50:18.230578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:18.230558    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 06:50:22.264332 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.264273    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:50:22.352818 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.352793    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.352954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.352881    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.352954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.352918    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.353072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.352955    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.353072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353002    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.353175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353140    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.353231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353184    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.353305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353256    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.353438 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353179    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:22.353538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353262    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:22.353538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353357    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:22.353538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353367    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:22.353701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353634    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data\") pod \"be32ed1d-772c-4a98-926a-653cb28329c1\" (UID: \"be32ed1d-772c-4a98-926a-653cb28329c1\") "
Nov 01 06:50:22.354020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.353999    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:22.354176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.354156    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.354247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.354182    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be32ed1d-772c-4a98-926a-653cb28329c1-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.354247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.354197    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.354247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.354212    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.354247 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.354225    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.355236 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.355210    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:22.355333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.355289    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:50:22.355622 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.355603    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:22.355722 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.355702    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2" pod "be32ed1d-772c-4a98-926a-653cb28329c1" (UID: "be32ed1d-772c-4a98-926a-653cb28329c1"). InnerVolumeSpecName "kube-api-access-hv6j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:22.455198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.455167    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be32ed1d-772c-4a98-926a-653cb28329c1-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.455198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.455195    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6j2\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-kube-api-access-hv6j2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.455382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.455206    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be32ed1d-772c-4a98-926a-653cb28329c1-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:22.455382 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:22.455215    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be32ed1d-772c-4a98-926a-653cb28329c1-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:23.035258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.035176    4468 generic.go:356] "Generic (PLEG): container finished" podID="be32ed1d-772c-4a98-926a-653cb28329c1" containerID="858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b" exitCode=0
Nov 01 06:50:23.035258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.035224    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" event={"ID":"be32ed1d-772c-4a98-926a-653cb28329c1","Type":"ContainerDied","Data":"858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b"}
Nov 01 06:50:23.035258 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.035231    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"
Nov 01 06:50:23.035521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.035276    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc" event={"ID":"be32ed1d-772c-4a98-926a-653cb28329c1","Type":"ContainerDied","Data":"e40e03dd4c8a4de94de7439956d3fe0670c3f9a7f26b575c370345bf49ea35ba"}
Nov 01 06:50:23.035521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.035306    4468 scope.go:122] "RemoveContainer" containerID="858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b"
Nov 01 06:50:23.045090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.045055    4468 scope.go:122] "RemoveContainer" containerID="7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"
Nov 01 06:50:23.053657 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.053637    4468 scope.go:122] "RemoveContainer" containerID="858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b"
Nov 01 06:50:23.053933 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:50:23.053915    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b\": container with ID starting with 858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b not found: ID does not exist" containerID="858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b"
Nov 01 06:50:23.053983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.053943    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b"} err="failed to get container status \"858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b\": rpc error: code = NotFound desc = could not find container \"858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b\": container with ID starting with 858e042b368ffd8371741c2b6b9f4027b50c65a907207d3defbf7921a4e92e2b not found: ID does not exist"
Nov 01 06:50:23.053983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.053962    4468 scope.go:122] "RemoveContainer" containerID="7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"
Nov 01 06:50:23.054219 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:50:23.054201    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8\": container with ID starting with 7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8 not found: ID does not exist" containerID="7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"
Nov 01 06:50:23.054259 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.054228    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8"} err="failed to get container status \"7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8\": rpc error: code = NotFound desc = could not find container \"7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8\": container with ID starting with 7b7f5f47da97b7516699fe2b8461a688fb0efa120bd321d9af17a3f5caa569a8 not found: ID does not exist"
Nov 01 06:50:23.248986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.248952    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"]
Nov 01 06:50:23.298679 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.298651    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8a9760-openshift-default-57f6f4564c-pcwsc"]
Nov 01 06:50:23.723266 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723181    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"]
Nov 01 06:50:23.723811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723792    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy"
Nov 01 06:50:23.723811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723812    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy"
Nov 01 06:50:23.723980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723826    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy"
Nov 01 06:50:23.723980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723833    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy"
Nov 01 06:50:23.723980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723846    4468 cpu_manager.go:414] "containerMap: removing container" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy"
Nov 01 06:50:23.723980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723853    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy"
Nov 01 06:50:23.723980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723880    4468 cpu_manager.go:414] "containerMap: removing container" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy"
Nov 01 06:50:23.723980 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.723887    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy"
Nov 01 06:50:23.724262 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.724096    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy"
Nov 01 06:50:23.724262 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.724125    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy"
Nov 01 06:50:23.724262 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.724135    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d705c95-3058-430e-8915-ef05b0bb6175" containerName="istio-proxy"
Nov 01 06:50:23.728891 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.728869    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.769568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.769537    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.769568 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.769572    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.769787 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.769763    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.769837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.769803    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.769914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.769897    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.769965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.769934    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kns7b\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.770020 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.770005    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.770109 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.770094    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.770161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.770153    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.787300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.787262    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"]
Nov 01 06:50:23.806440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.806414    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-302ba0-openshift-default-dockercfg-gh4kk\""
Nov 01 06:50:23.871252 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.871217    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.871563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.871542    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.872080 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.871917    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.872500 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872440    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.872621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872515    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.872743 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872725    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.872842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872763    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kns7b\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.872842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872803    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.873022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872875    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.873257 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.873235    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.873379 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.871949    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.873576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.873498    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.873576 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.872137    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.874174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.874149    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.880399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.880356    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.918069 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.918032    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.942097 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.939768    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kns7b\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:23.956563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:23.942821    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:24.047876 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.047848    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:24.482827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.482747    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"]
Nov 01 06:50:24.505101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.505071    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:50:24.521309 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:50:24.521278    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f695329_a1fa_49f4_83bc_0b64fca90d7d.slice/crio-910c48ebef881453d7c257386fb43a8cbc2cda7a20c06e826e5d89a9946fd282 WatchSource:0}: Error finding container 910c48ebef881453d7c257386fb43a8cbc2cda7a20c06e826e5d89a9946fd282: Status 404 returned error can't find the container with id 910c48ebef881453d7c257386fb43a8cbc2cda7a20c06e826e5d89a9946fd282
Nov 01 06:50:24.523572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.523540    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:50:24.523661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.523601    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:50:24.523661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.523630    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:50:24.590978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:24.590954    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" path="/var/lib/kubelet/pods/be32ed1d-772c-4a98-926a-653cb28329c1/volumes"
Nov 01 06:50:25.069072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:25.069031    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" event={"ID":"1f695329-a1fa-49f4-83bc-0b64fca90d7d","Type":"ContainerStarted","Data":"9c49963b9ac9318f0e7801264fd172706e8c232d9006f7e84a865645e01c1273"}
Nov 01 06:50:25.069287 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:25.069101    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" event={"ID":"1f695329-a1fa-49f4-83bc-0b64fca90d7d","Type":"ContainerStarted","Data":"910c48ebef881453d7c257386fb43a8cbc2cda7a20c06e826e5d89a9946fd282"}
Nov 01 06:50:25.188498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:25.188417    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podStartSLOduration=2.188398875 podStartE2EDuration="2.188398875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:25.185948494 +0000 UTC m=+2436.753755334" watchObservedRunningTime="2026-11-01 06:50:25.188398875 +0000 UTC m=+2436.756205705"
Nov 01 06:50:26.048218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:26.048174    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:26.054622 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:26.054583    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:26.078689 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:26.078661    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:26.079811 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:26.079786    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:50:28.576618 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:28.576582    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-g2829 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:50:28.624277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:28.624233    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:50:32.950903 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:50:32.950862    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:50:45.147954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:45.147922    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:50:51.213153 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:51.213123    4468 scope.go:122] "RemoveContainer" containerID="db246e1ec6a3971a8710e9fd472ba104f9d942398a84861ff1c2d18ed6109a49"
Nov 01 06:50:51.222762 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:51.222744    4468 scope.go:122] "RemoveContainer" containerID="9e5bb95bd412380f2adf8c8d5501721294701446eec789ba2dde8cc20967fde2"
Nov 01 06:50:51.235134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:51.231594    4468 scope.go:122] "RemoveContainer" containerID="435e800e4e2aad4c69ae2366081420d8317188cb3b45254a0d3c11a444dfa385"
Nov 01 06:50:52.329800 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.329768    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"]
Nov 01 06:50:52.332533 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.331327    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="be32ed1d-772c-4a98-926a-653cb28329c1" containerName="istio-proxy"
Nov 01 06:50:52.337732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.337704    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.390577 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.390548    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"]
Nov 01 06:50:52.435446 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.435405    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:50:52.435446 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.435416    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:50:52.435642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.435451    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:50:52.440521 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.440505    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:50:52.499656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.499626    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.499793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.499664    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.499793 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.499766    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttncz\" (UniqueName: \"kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.601157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.601086    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.601157 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.601121    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.601308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.601180    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttncz\" (UniqueName: \"kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.601844 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.601826    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.603637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.603611    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.657894 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.657867    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttncz\" (UniqueName: \"kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz\") pod \"maas-api-revoke-sub-6a8702c6-lvdwd\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:52.953632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:52.953558    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:53.326396 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:53.326368    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"]
Nov 01 06:50:53.326990 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:50:53.326967    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1b84c7_e06a_45cb_80e8_d42acff6e581.slice/crio-0d37b98eaecedbd825e340797f0b6ba41fd01660f6bb4fada44a1556b24535ea WatchSource:0}: Error finding container 0d37b98eaecedbd825e340797f0b6ba41fd01660f6bb4fada44a1556b24535ea: Status 404 returned error can't find the container with id 0d37b98eaecedbd825e340797f0b6ba41fd01660f6bb4fada44a1556b24535ea
Nov 01 06:50:53.354965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:53.354933    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd" event={"ID":"8a1b84c7-e06a-45cb-80e8-d42acff6e581","Type":"ContainerStarted","Data":"0d37b98eaecedbd825e340797f0b6ba41fd01660f6bb4fada44a1556b24535ea"}
Nov 01 06:50:54.362737 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:54.362700    4468 generic.go:356] "Generic (PLEG): container finished" podID="8a1b84c7-e06a-45cb-80e8-d42acff6e581" containerID="c1eb43e5a5f1f4c2591430db36fbc1ebfd280759874e795ad72ae035f405bc2c" exitCode=0
Nov 01 06:50:54.363437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:54.362742    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd" event={"ID":"8a1b84c7-e06a-45cb-80e8-d42acff6e581","Type":"ContainerDied","Data":"c1eb43e5a5f1f4c2591430db36fbc1ebfd280759874e795ad72ae035f405bc2c"}
Nov 01 06:50:55.510837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.510808    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:55.637149 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.637057    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz\" (UniqueName: \"kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz\") pod \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") "
Nov 01 06:50:55.637319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.637174    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca\") pod \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") "
Nov 01 06:50:55.637319 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.637248    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca\") pod \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\" (UID: \"8a1b84c7-e06a-45cb-80e8-d42acff6e581\") "
Nov 01 06:50:55.637676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.637648    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca" pod "8a1b84c7-e06a-45cb-80e8-d42acff6e581" (UID: "8a1b84c7-e06a-45cb-80e8-d42acff6e581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:55.637942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.637907    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:55.639399 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.639371    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca" pod "8a1b84c7-e06a-45cb-80e8-d42acff6e581" (UID: "8a1b84c7-e06a-45cb-80e8-d42acff6e581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:50:55.639523 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.639448    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz" pod "8a1b84c7-e06a-45cb-80e8-d42acff6e581" (UID: "8a1b84c7-e06a-45cb-80e8-d42acff6e581"). InnerVolumeSpecName "kube-api-access-ttncz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:55.738907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.738878    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttncz\" (UniqueName: \"kubernetes.io/projected/8a1b84c7-e06a-45cb-80e8-d42acff6e581-kube-api-access-ttncz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:55.738907 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:55.738904    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b84c7-e06a-45cb-80e8-d42acff6e581-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:56.376355 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:56.376324    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"
Nov 01 06:50:56.376589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:56.376353    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd" event={"ID":"8a1b84c7-e06a-45cb-80e8-d42acff6e581","Type":"ContainerDied","Data":"0d37b98eaecedbd825e340797f0b6ba41fd01660f6bb4fada44a1556b24535ea"}
Nov 01 06:50:56.376589 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:50:56.376401    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d37b98eaecedbd825e340797f0b6ba41fd01660f6bb4fada44a1556b24535ea"
Nov 01 06:51:02.950142 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:51:02.950055    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:51:12.255445 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:12.255411    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:51:15.781956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:15.781574    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-p4lr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:51:15.781956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:15.781655    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:51:30.181360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:30.181327    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"]
Nov 01 06:51:30.181945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:30.181606    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" containerID="cri-o://18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203" gracePeriod=30
Nov 01 06:51:31.770358 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.770314    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"]
Nov 01 06:51:31.771243 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.771222    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8a1b84c7-e06a-45cb-80e8-d42acff6e581" containerName="revoke-keys"
Nov 01 06:51:31.771297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.771247    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1b84c7-e06a-45cb-80e8-d42acff6e581" containerName="revoke-keys"
Nov 01 06:51:31.771408 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.771396    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1b84c7-e06a-45cb-80e8-d42acff6e581" containerName="revoke-keys"
Nov 01 06:51:31.776025 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.775993    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.862075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.862038    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"]
Nov 01 06:51:31.870852 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.870790    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.871178 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.871158    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.871411 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.871395    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.871628 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.871600    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.871815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.871799    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.872072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.872054    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.872231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.872217    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.872400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.872371    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7h2\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.872572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.872558    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.897672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.897636    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-28c598-openshift-default-dockercfg-z7ljk\""
Nov 01 06:51:31.974390 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974358    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974400    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974453    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974501    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974537    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974615    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974631    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974649    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7h2\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974670    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974797    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.974951 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.974914    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.975174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.975027    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.975174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.975052    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.975450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.975381    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.976995 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.976975    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:31.977130 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:31.977115    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:32.031659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.031585    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:32.031659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.031606    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7h2\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:32.093113 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.093077    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:32.532422 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.532392    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"]
Nov 01 06:51:32.533310 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:51:32.533284    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adc3f32_2220_4c25_849e_c873a4ff8c2c.slice/crio-83e1dfea4f7ffaeff7174d07a4d47cd643271f475a8e33d06c8c7b9ee71eeede WatchSource:0}: Error finding container 83e1dfea4f7ffaeff7174d07a4d47cd643271f475a8e33d06c8c7b9ee71eeede: Status 404 returned error can't find the container with id 83e1dfea4f7ffaeff7174d07a4d47cd643271f475a8e33d06c8c7b9ee71eeede
Nov 01 06:51:32.536031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.535996    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:32.536111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.536070    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:32.536153 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.536108    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:32.912061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.912029    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" event={"ID":"0adc3f32-2220-4c25-849e-c873a4ff8c2c","Type":"ContainerStarted","Data":"944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"}
Nov 01 06:51:32.912061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:32.912077    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" event={"ID":"0adc3f32-2220-4c25-849e-c873a4ff8c2c","Type":"ContainerStarted","Data":"83e1dfea4f7ffaeff7174d07a4d47cd643271f475a8e33d06c8c7b9ee71eeede"}
Nov 01 06:51:32.947968 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:51:32.947924    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:51:33.030978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:33.030909    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podStartSLOduration=2.030888296 podStartE2EDuration="2.030888296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:33.027806408 +0000 UTC m=+2504.595613248" watchObservedRunningTime="2026-11-01 06:51:33.030888296 +0000 UTC m=+2504.598695127"
Nov 01 06:51:33.093198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:33.093165    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:33.098788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:33.098764    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:33.919617 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:33.919585    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:33.920679 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:33.920640    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:51:34.861302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:34.861261    4468 patch_prober.go:29] interesting pod/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:51:34.861527 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:34.861346    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:51:35.550753 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.550724    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:51:35.644932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.644859    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645014    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645055    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645140 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645107    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645165    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645246    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645337 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645273    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645359    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645411    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs\") pod \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\" (UID: \"8161e4f0-456c-427f-ae79-b2fa97b5ea6d\") "
Nov 01 06:51:35.645518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645415    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:35.645988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645780    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:35.646191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.645938    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:35.646191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.646129    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:35.646191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.646135    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.646191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.646158    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.646191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.646172    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.646422 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.646219    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:35.647495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.647436    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:35.647662 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.647591    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:35.648356 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.648321    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:51:35.648988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.648904    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw" pod "8161e4f0-456c-427f-ae79-b2fa97b5ea6d" (UID: "8161e4f0-456c-427f-ae79-b2fa97b5ea6d"). InnerVolumeSpecName "kube-api-access-mfxdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:35.752524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.750387    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.752524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.750428    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.752524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.750444    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxdw\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-kube-api-access-mfxdw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.752524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.750460    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.752524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.750492    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.752524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.750504    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8161e4f0-456c-427f-ae79-b2fa97b5ea6d-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.934123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.934088    4468 generic.go:356] "Generic (PLEG): container finished" podID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerID="18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203" exitCode=0
Nov 01 06:51:35.934318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.934143    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"
Nov 01 06:51:35.934318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.934179    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" event={"ID":"8161e4f0-456c-427f-ae79-b2fa97b5ea6d","Type":"ContainerDied","Data":"18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203"}
Nov 01 06:51:35.934318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.934247    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm" event={"ID":"8161e4f0-456c-427f-ae79-b2fa97b5ea6d","Type":"ContainerDied","Data":"695a89c45d058374ccbc11fd1c1fd11add7f23551bc2f009562dea084b209a51"}
Nov 01 06:51:35.934318 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.934282    4468 scope.go:122] "RemoveContainer" containerID="18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203"
Nov 01 06:51:35.944986 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.944964    4468 scope.go:122] "RemoveContainer" containerID="857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"
Nov 01 06:51:35.954567 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.954549    4468 scope.go:122] "RemoveContainer" containerID="18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203"
Nov 01 06:51:35.954823 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:51:35.954802    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203\": container with ID starting with 18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203 not found: ID does not exist" containerID="18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203"
Nov 01 06:51:35.954865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.954835    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203"} err="failed to get container status \"18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203\": rpc error: code = NotFound desc = could not find container \"18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203\": container with ID starting with 18b88e34beb5a2e58668cb12d58289e5da8639f6310583300f242628ad62c203 not found: ID does not exist"
Nov 01 06:51:35.954865 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.954861    4468 scope.go:122] "RemoveContainer" containerID="857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"
Nov 01 06:51:35.955088 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:51:35.955072    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466\": container with ID starting with 857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466 not found: ID does not exist" containerID="857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"
Nov 01 06:51:35.955125 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:35.955095    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466"} err="failed to get container status \"857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466\": rpc error: code = NotFound desc = could not find container \"857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466\": container with ID starting with 857729577dae96665fd8bdb70487202ff2a6024fadafb7aa2cd86e85158b3466 not found: ID does not exist"
Nov 01 06:51:36.098874 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:36.098844    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"]
Nov 01 06:51:36.155418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:36.155383    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d855db-openshift-default-577cbd8699-6djwm"]
Nov 01 06:51:36.590029 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:36.589991    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" path="/var/lib/kubelet/pods/8161e4f0-456c-427f-ae79-b2fa97b5ea6d/volumes"
Nov 01 06:51:39.335571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.335534    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"]
Nov 01 06:51:39.336260 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.336238    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy"
Nov 01 06:51:39.336369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.336262    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy"
Nov 01 06:51:39.336369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.336301    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy"
Nov 01 06:51:39.336369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.336309    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy"
Nov 01 06:51:39.336573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.336493    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy"
Nov 01 06:51:39.341638 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.341616    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.439015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.438976    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"]
Nov 01 06:51:39.452974 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.452901    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a3e841-openshift-default-dockercfg-tk8x2\""
Nov 01 06:51:39.550774 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.550743    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.550959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.550778    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.550959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.550812    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.550959 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.550919    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.551107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.550974    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rtc\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.551107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.550995    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.551217 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.551191    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.551293 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.551260    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.551335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.551304    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652051    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rtc\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652092    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652193    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652333 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652240    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652433 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652354    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652460    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652563 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652505    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652624 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652563    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652624 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652597    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.652730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.652683    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.653175 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.653150    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.653307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.653172    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.653307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.653207    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.653307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.653238    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.654916 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.654893    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.655009 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.654935    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.715398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.715363    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.715595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.715562    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rtc\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc\") pod \"e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") " pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:39.954504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:39.954387    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:40.385672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:40.385637    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"]
Nov 01 06:51:40.386126 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:51:40.386078    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9751b1_0e80_47d2_a676_91cb0edd19fa.slice/crio-5aae717655111bf00f251037bb12b901de7f0d137486a41af3bd0bb7b1c88393 WatchSource:0}: Error finding container 5aae717655111bf00f251037bb12b901de7f0d137486a41af3bd0bb7b1c88393: Status 404 returned error can't find the container with id 5aae717655111bf00f251037bb12b901de7f0d137486a41af3bd0bb7b1c88393
Nov 01 06:51:40.389536 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:40.389498    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:40.389646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:40.389575    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:40.389646 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:40.389608    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:40.967593 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:40.967549    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" event={"ID":"ae9751b1-0e80-47d2-a676-91cb0edd19fa","Type":"ContainerStarted","Data":"d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"}
Nov 01 06:51:40.967783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:40.967618    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" event={"ID":"ae9751b1-0e80-47d2-a676-91cb0edd19fa","Type":"ContainerStarted","Data":"5aae717655111bf00f251037bb12b901de7f0d137486a41af3bd0bb7b1c88393"}
Nov 01 06:51:41.087121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:41.087073    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podStartSLOduration=2.087058765 podStartE2EDuration="2.087058765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:41.083064387 +0000 UTC m=+2512.650871218" watchObservedRunningTime="2026-11-01 06:51:41.087058765 +0000 UTC m=+2512.654865593"
Nov 01 06:51:41.955565 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:41.955533    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:41.962188 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:41.962156    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:41.975352 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:41.975319    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:41.976496 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:41.976456    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:51:51.273956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:51.273925    4468 scope.go:122] "RemoveContainer" containerID="dfa48a4894c11bfa4d431de6f3597ab41bace52350ed74210ac69a216a0d9f22"
Nov 01 06:51:56.239398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:56.238305    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": read tcp 10.133.0.2:47522->10.133.0.159:15021: read: connection reset by peer" start-of-body=
Nov 01 06:51:56.239398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:56.238362    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": read tcp 10.133.0.2:47522->10.133.0.159:15021: read: connection reset by peer"
Nov 01 06:51:57.100164 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:57.100131    4468 generic.go:356] "Generic (PLEG): container finished" podID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerID="9c49963b9ac9318f0e7801264fd172706e8c232d9006f7e84a865645e01c1273" exitCode=0
Nov 01 06:51:57.100378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:57.100208    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" event={"ID":"1f695329-a1fa-49f4-83bc-0b64fca90d7d","Type":"ContainerDied","Data":"9c49963b9ac9318f0e7801264fd172706e8c232d9006f7e84a865645e01c1273"}
Nov 01 06:51:57.100682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:57.100665    4468 scope.go:122] "RemoveContainer" containerID="9c49963b9ac9318f0e7801264fd172706e8c232d9006f7e84a865645e01c1273"
Nov 01 06:51:57.102181 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:57.102152    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:57.102279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:57.102221    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:57.102279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:57.102263    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:51:58.135692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:58.135651    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" event={"ID":"1f695329-a1fa-49f4-83bc-0b64fca90d7d","Type":"ContainerStarted","Data":"c041c49ff9f6c0b85c32b51e61380d812b8500d3792b41647531e61b6d5682bc"}
Nov 01 06:51:59.048572 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:59.048525    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:51:59.048831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:59.048804    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 06:51:59.048909 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:51:59.048870    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 06:52:00.048883 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:00.048846    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 06:52:00.049279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:00.048933    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 06:52:01.048414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:01.048380    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 06:52:01.048715 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:01.048451    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 06:52:02.048580 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:02.048549    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 06:52:02.048996 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:02.048618    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 06:52:02.952950 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:52:02.952912    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:52:03.048302 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:03.048260    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 06:52:03.048508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:03.048334    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 06:52:04.052450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:04.052420    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:52:04.052900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:04.052830    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:52:04.053671 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:04.053649    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 06:52:07.566816 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:07.566785    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:52:11.045714 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:11.045675    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"]
Nov 01 06:52:11.095097 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:11.095068    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89b919e0-9lp42"]
Nov 01 06:52:12.589926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:12.589894    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60a1dd4-4405-4e7d-9b33-4b15513682bb" path="/var/lib/kubelet/pods/a60a1dd4-4405-4e7d-9b33-4b15513682bb/volumes"
Nov 01 06:52:14.045595 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:14.045561    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"]
Nov 01 06:52:14.094092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:14.094065    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe5d85e-m5gqd"]
Nov 01 06:52:14.590114 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:14.590079    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852ac969-e217-4fb0-9799-ecafbf65ee93" path="/var/lib/kubelet/pods/852ac969-e217-4fb0-9799-ecafbf65ee93/volumes"
Nov 01 06:52:22.045026 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:22.044996    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"]
Nov 01 06:52:22.094245 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:22.094221    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00be7374-2qtjv"]
Nov 01 06:52:22.590101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:22.590070    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97e613f-dd09-4584-b061-9b9274c4498a" path="/var/lib/kubelet/pods/f97e613f-dd09-4584-b061-9b9274c4498a/volumes"
Nov 01 06:52:26.044607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:26.044571    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"]
Nov 01 06:52:26.093832 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:26.093799    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf0148a-sks4l"]
Nov 01 06:52:26.590414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:26.590376    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d559858-78d8-4d3f-9034-54403599d2a9" path="/var/lib/kubelet/pods/7d559858-78d8-4d3f-9034-54403599d2a9/volumes"
Nov 01 06:52:30.956799 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:30.956711    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:52:32.947759 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:52:32.947713    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:52:38.046046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:38.046009    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"]
Nov 01 06:52:38.095428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:38.095394    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cab84f-h7n6x"]
Nov 01 06:52:38.590459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:38.590424    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e67296-aac8-4069-a2f1-3f8e696d5ce4" path="/var/lib/kubelet/pods/72e67296-aac8-4069-a2f1-3f8e696d5ce4/volumes"
Nov 01 06:52:40.841790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.841739    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"]
Nov 01 06:52:40.842808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.842782    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8161e4f0-456c-427f-ae79-b2fa97b5ea6d" containerName="istio-proxy"
Nov 01 06:52:40.847840 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.847813    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:40.897678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.897645    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"]
Nov 01 06:52:40.921637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.921605    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxt7r\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:40.921637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.921648    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:40.921948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.921926    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:40.922044 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.922030    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:40.942642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942612    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"e2e-cfl-6b3232-provider-cred\""
Nov 01 06:52:40.942819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942617    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"agc-cred-dcba5d3eb853\""
Nov 01 06:52:40.942819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942653    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"payload-processing-external-model-e2e--b10087ce-dockercfg-crgnv\""
Nov 01 06:52:40.942819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942663    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-caf74d\""
Nov 01 06:52:40.942819 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942684    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"kube-root-ca.crt\""
Nov 01 06:52:40.943022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942694    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"openshift-service-ca.crt\""
Nov 01 06:52:40.943022 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:40.942770    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"routing-overlay\""
Nov 01 06:52:41.023459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.023427    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxt7r\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.023459 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.023480    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.023692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.023631    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.023692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.023676    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.024218 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.024195    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.024503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.024263    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.026212 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.026190    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.045531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.045507    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"]
Nov 01 06:52:41.052426 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.052403    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbbc6d9-lp7xr"]
Nov 01 06:52:41.080152 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.080121    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxt7r\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.167812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.167738    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:41.544127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.544093    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"]
Nov 01 06:52:41.545231 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:52:41.545201    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fa6e73_5678_4f31_b603_71204e8e9ea2.slice/crio-bcd50a858a5553dc10ea299ced9528e2f82d61bf115b1a54ccbf0251e175ac4e WatchSource:0}: Error finding container bcd50a858a5553dc10ea299ced9528e2f82d61bf115b1a54ccbf0251e175ac4e: Status 404 returned error can't find the container with id bcd50a858a5553dc10ea299ced9528e2f82d61bf115b1a54ccbf0251e175ac4e
Nov 01 06:52:41.572647 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.572623    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"]
Nov 01 06:52:41.578161 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.578140    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.689514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.656889    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6ba6e5-openshift-default-dockercfg-6c4gs\""
Nov 01 06:52:41.689514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.676620    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"]
Nov 01 06:52:41.733728 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.733683    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.734064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.734036    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.734323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.734299    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.777195 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.777157    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.777552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.777514    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.777810 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.777783    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnlw\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.779716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.779694    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.779978 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.779963    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.780104 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.780090    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881331    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881407    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881432    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881507    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881582    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881654    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881686    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881715    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881805    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnlw\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.881837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881822    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.882276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881899    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.882276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.881996    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.882276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.882050    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.882276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.882245    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.884198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.884172    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.884300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.884290    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.939851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.939812    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:41.940006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:41.939936    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnlw\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw\") pod \"e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") " pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:42.231812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.231735    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:42.431966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.431921    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4" event={"ID":"10fa6e73-5678-4f31-b603-71204e8e9ea2","Type":"ContainerStarted","Data":"d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93"}
Nov 01 06:52:42.432125 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.431980    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4" event={"ID":"10fa6e73-5678-4f31-b603-71204e8e9ea2","Type":"ContainerStarted","Data":"bcd50a858a5553dc10ea299ced9528e2f82d61bf115b1a54ccbf0251e175ac4e"}
Nov 01 06:52:42.432170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.432121    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:42.543814 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.543710    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4" podStartSLOduration=2.543695909 podStartE2EDuration="2.543695909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:52:42.543451021 +0000 UTC m=+2574.111257863" watchObservedRunningTime="2026-11-01 06:52:42.543695909 +0000 UTC m=+2574.111502738"
Nov 01 06:52:42.590418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.590387    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c874901-4cfa-41af-8be0-063c5e87f621" path="/var/lib/kubelet/pods/3c874901-4cfa-41af-8be0-063c5e87f621/volumes"
Nov 01 06:52:42.708380 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.708348    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"]
Nov 01 06:52:42.709155 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:52:42.709128    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bc3aa9_90af_4e8c_bb37_e614ca82d170.slice/crio-0c50c5ac4e5dbf90d503d960aff94b3fe7e477943228ec4cd17f0bc0c71e86f2 WatchSource:0}: Error finding container 0c50c5ac4e5dbf90d503d960aff94b3fe7e477943228ec4cd17f0bc0c71e86f2: Status 404 returned error can't find the container with id 0c50c5ac4e5dbf90d503d960aff94b3fe7e477943228ec4cd17f0bc0c71e86f2
Nov 01 06:52:42.712344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.712287    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:52:42.712450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.712393    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:52:42.712450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:42.712440    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:52:43.440749 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:43.440706    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" event={"ID":"b3bc3aa9-90af-4e8c-bb37-e614ca82d170","Type":"ContainerStarted","Data":"5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"}
Nov 01 06:52:43.441211 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:43.440804    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" event={"ID":"b3bc3aa9-90af-4e8c-bb37-e614ca82d170","Type":"ContainerStarted","Data":"0c50c5ac4e5dbf90d503d960aff94b3fe7e477943228ec4cd17f0bc0c71e86f2"}
Nov 01 06:52:43.556170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:43.556113    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" podStartSLOduration=2.556099606 podStartE2EDuration="2.556099606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:52:43.555703658 +0000 UTC m=+2575.123510488" watchObservedRunningTime="2026-11-01 06:52:43.556099606 +0000 UTC m=+2575.123906434"
Nov 01 06:52:44.232297 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:44.232258    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:44.238131 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:44.238105    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:44.460196 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:44.460143    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:44.461049 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:44.461027    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:52:53.047282 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:53.047238    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"]
Nov 01 06:52:53.097013 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:53.096978    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7e13e3-5kgzh"]
Nov 01 06:52:53.099751 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:53.099736    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"]
Nov 01 06:52:53.102233 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:53.102208    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7f3360-fqdvl"]
Nov 01 06:52:53.443884 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:53.443807    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:52:54.590494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:54.590445    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1" path="/var/lib/kubelet/pods/5ab93a96-2ebd-4ce6-9442-ca3e6c78bda1/volumes"
Nov 01 06:52:54.591103 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:52:54.591081    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7" path="/var/lib/kubelet/pods/8c5a471d-d3c8-4bb3-acfb-b7e008eb94b7/volumes"
Nov 01 06:53:02.952194 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:53:02.952154    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:53:07.045716 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:07.045677    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"]
Nov 01 06:53:07.095064 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:07.095028    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c8871b-qmtdx"]
Nov 01 06:53:07.854301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:07.854267    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:53:08.080558 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:08.080521    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"]
Nov 01 06:53:08.093491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:08.088217    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0526811-mnz4b"]
Nov 01 06:53:08.630506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:08.615874    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399fe1f3-0227-458b-9c08-c8c63e4ce67b" path="/var/lib/kubelet/pods/399fe1f3-0227-458b-9c08-c8c63e4ce67b/volumes"
Nov 01 06:53:08.631564 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:08.631529    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe71916-e0c6-4b6c-99d1-0449cb9b2a13" path="/var/lib/kubelet/pods/cbe71916-e0c6-4b6c-99d1-0449cb9b2a13/volumes"
Nov 01 06:53:18.712781 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.712752    4468 generic.go:356] "Generic (PLEG): container finished" podID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerID="944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047" exitCode=0
Nov 01 06:53:18.713230 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.712825    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" event={"ID":"0adc3f32-2220-4c25-849e-c873a4ff8c2c","Type":"ContainerDied","Data":"944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"}
Nov 01 06:53:18.713301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.713277    4468 scope.go:122] "RemoveContainer" containerID="944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"
Nov 01 06:53:18.714643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.714620    4468 generic.go:356] "Generic (PLEG): container finished" podID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerID="d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f" exitCode=0
Nov 01 06:53:18.714719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.714687    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" event={"ID":"ae9751b1-0e80-47d2-a676-91cb0edd19fa","Type":"ContainerDied","Data":"d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"}
Nov 01 06:53:18.714954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.714927    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:53:18.715011 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.714998    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:53:18.715046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.715015    4468 scope.go:122] "RemoveContainer" containerID="d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"
Nov 01 06:53:18.715046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.715033    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:53:18.716652 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.716628    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:53:18.716719 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.716704    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:53:18.716764 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.716750    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:53:18.920034 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:18.919967    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:53:19.048501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.048134    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"]
Nov 01 06:53:19.050806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.050779    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bb4d35-t589v"]
Nov 01 06:53:19.759372 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.759330    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" event={"ID":"0adc3f32-2220-4c25-849e-c873a4ff8c2c","Type":"ContainerStarted","Data":"4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812"}
Nov 01 06:53:19.762824 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.762229    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" event={"ID":"ae9751b1-0e80-47d2-a676-91cb0edd19fa","Type":"ContainerStarted","Data":"fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd"}
Nov 01 06:53:19.954906 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.954864    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:53:19.955222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.955195    4468 patch_prober.go:29] interesting pod/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:53:19.955271 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:19.955247    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:53:20.059129 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.059082    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"]
Nov 01 06:53:20.061682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.061654    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7747dd-v5bz8"]
Nov 01 06:53:20.094005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.093961    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:53:20.095147 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.095118    4468 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 06:53:20.095275 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.095176    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 06:53:20.590398 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.590367    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e008f13-8f3e-4e9f-9f31-1350c1a6a340" path="/var/lib/kubelet/pods/7e008f13-8f3e-4e9f-9f31-1350c1a6a340/volumes"
Nov 01 06:53:20.591031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.591015    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c638b09b-ab4e-4266-9be6-fdb3d2f09f74" path="/var/lib/kubelet/pods/c638b09b-ab4e-4266-9be6-fdb3d2f09f74/volumes"
Nov 01 06:53:20.955917 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.955827    4468 patch_prober.go:29] interesting pod/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:53:20.956310 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:20.955913    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:53:21.093709 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:21.093670    4468 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 06:53:21.093900 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:21.093730    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 06:53:21.955079 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:21.955032    4468 patch_prober.go:29] interesting pod/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:53:21.955309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:21.955117    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:53:22.093777 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:22.093734    4468 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 06:53:22.094279 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:22.093805    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 06:53:22.955687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:22.955639    4468 patch_prober.go:29] interesting pod/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:53:22.955887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:22.955720    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:53:23.093879 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:23.093831    4468 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 06:53:23.094309 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:23.093892    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 06:53:23.955440 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:23.955399    4468 patch_prober.go:29] interesting pod/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:53:23.955698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:23.955459    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:53:24.093602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:24.093554    4468 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 01 06:53:24.093812 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:24.093661    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 01 06:53:24.959952 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:24.959915    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:53:24.960368 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:24.960299    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:53:24.961046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:24.961028    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:53:25.098835 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:25.098799    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:53:25.099231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:25.099204    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:53:25.099911 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:25.099890    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 06:53:32.947836 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:53:32.947799    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:53:51.316725 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.316683    4468 scope.go:122] "RemoveContainer" containerID="74adcbd378907201360a4612f2732c6f51e0a158eaea480efe4181b0a264b0e7"
Nov 01 06:53:51.326982 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.326958    4468 scope.go:122] "RemoveContainer" containerID="0d6785b3f4444ede9395cd450c4ed35a812720698ade606d810f856260713909"
Nov 01 06:53:51.339162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.338632    4468 scope.go:122] "RemoveContainer" containerID="de5078183276e1187f48837f858e4d253bbf434019b6ab30f1fabfd5a0cc79b1"
Nov 01 06:53:51.347889 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.347870    4468 scope.go:122] "RemoveContainer" containerID="db9fff4abc0733210756cd2e080cfe3e4662f16df3dfbabaf1c178d751c60aff"
Nov 01 06:53:51.356515 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.356491    4468 scope.go:122] "RemoveContainer" containerID="cba9453e7949cb1c95778a022c4c99f830e1b4a0070cf2dda29420d68d6c84b6"
Nov 01 06:53:51.365277 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.365257    4468 scope.go:122] "RemoveContainer" containerID="8b675135e75ef73e0de88de1cd1a188ec3c287dde04e5b08d3ea9e97c784b621"
Nov 01 06:53:51.373857 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.373836    4468 scope.go:122] "RemoveContainer" containerID="d3db45ef171dd42032c56ad20ef13d335c5b39a88e3a74e996daafc6c229e87e"
Nov 01 06:53:51.382330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:53:51.382310    4468 scope.go:122] "RemoveContainer" containerID="505332a4b2053a44ef8a8101abf9feff30433724803122383b16a13f0411ecb3"
Nov 01 06:54:02.947602 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:54:02.947518    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:54:03.297817 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.297789    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"]
Nov 01 06:54:03.303954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.303933    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.319357 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.319335    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.319505 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.319396    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.319577 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.319529    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.319632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.319581    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglvz\" (UniqueName: \"kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.319684 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.319645    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.319736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.319714    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.358538 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.358497    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"]
Nov 01 06:54:03.404726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.404697    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-28c598\"/\"default-dockercfg-llxgf\""
Nov 01 06:54:03.404914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.404772    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-28c598\"/\"simulator-w3-28c598-kserve-self-signed-certs\""
Nov 01 06:54:03.405785 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.405761    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-28c598\"/\"openshift-service-ca.crt\""
Nov 01 06:54:03.405931 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.405802    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-28c598\"/\"kube-root-ca.crt\""
Nov 01 06:54:03.420637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.420606    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.420736 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.420695    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglvz\" (UniqueName: \"kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.420797 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.420750    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.420851 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.420815    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.420904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.420887    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.420963 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.420954    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.421170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.421149    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.421261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.421191    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.421261 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.421237    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.422987 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.422967    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.423400 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.423383    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.476890 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.476863    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglvz\" (UniqueName: \"kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz\") pod \"simulator-w3-28c598-kserve-5cfc5c4479-kd9np\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") " pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.617772 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.617688    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:03.989945 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:03.989910    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"]
Nov 01 06:54:03.991050 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:54:03.991024    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67527b2f_0dc7_4e36_ab04_6ccd371d2dff.slice/crio-342a67955825092847b2cf46d3b8070175edf3f364daa78f9d9c491b564d6a0e WatchSource:0}: Error finding container 342a67955825092847b2cf46d3b8070175edf3f364daa78f9d9c491b564d6a0e: Status 404 returned error can't find the container with id 342a67955825092847b2cf46d3b8070175edf3f364daa78f9d9c491b564d6a0e
Nov 01 06:54:04.067122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:04.066232    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" event={"ID":"67527b2f-0dc7-4e36-ab04-6ccd371d2dff","Type":"ContainerStarted","Data":"35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99"}
Nov 01 06:54:04.067122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:04.066279    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" event={"ID":"67527b2f-0dc7-4e36-ab04-6ccd371d2dff","Type":"ContainerStarted","Data":"342a67955825092847b2cf46d3b8070175edf3f364daa78f9d9c491b564d6a0e"}
Nov 01 06:54:04.181720 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:04.181663    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" podStartSLOduration=1.181645037 podStartE2EDuration="1.181645037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:04.179718833 +0000 UTC m=+2655.747525685" watchObservedRunningTime="2026-11-01 06:54:04.181645037 +0000 UTC m=+2655.749451872"
Nov 01 06:54:04.618206 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:04.618175    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:13.618687 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:13.618645    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:13.631492 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:13.631429    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:14.143834 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:14.143810    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 06:54:14.709060 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:14.709024    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:54:24.130574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.130539    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"]
Nov 01 06:54:24.134744 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.134718    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.191506 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.191456    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"]
Nov 01 06:54:24.226111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.226068    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.226299 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.226275    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.226540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.226494    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv498\" (UniqueName: \"kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.236918 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.236883    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:54:24.237066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.236971    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:54:24.237066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.236981    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:54:24.237171 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.237078    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:54:24.327692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.327655    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv498\" (UniqueName: \"kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.327863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.327768    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.327863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.327852    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.328701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.328677    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.330238 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.330215    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.387437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.387364    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv498\" (UniqueName: \"kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498\") pod \"maas-api-revoke-sub-3d4f828d-rs2fr\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.452461 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.452427    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:24.831345 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:24.831308    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"]
Nov 01 06:54:24.831611 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:54:24.831584    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71060936_60b8_4df6_bfdb_aa955b031810.slice/crio-e55e116b448298cee8b6ce762538759a97f25208b9f28d81cd160bdf20a41f54 WatchSource:0}: Error finding container e55e116b448298cee8b6ce762538759a97f25208b9f28d81cd160bdf20a41f54: Status 404 returned error can't find the container with id e55e116b448298cee8b6ce762538759a97f25208b9f28d81cd160bdf20a41f54
Nov 01 06:54:25.217450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:25.217410    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr" event={"ID":"71060936-60b8-4df6-bfdb-aa955b031810","Type":"ContainerStarted","Data":"62d6b8057462d932f2f4f81827477f6400252fc3900def4215a0fbcb6af2bcb1"}
Nov 01 06:54:25.217938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:25.217500    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr" event={"ID":"71060936-60b8-4df6-bfdb-aa955b031810","Type":"ContainerStarted","Data":"e55e116b448298cee8b6ce762538759a97f25208b9f28d81cd160bdf20a41f54"}
Nov 01 06:54:25.330489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:25.330265    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr" podStartSLOduration=1.330248699 podStartE2EDuration="1.330248699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:25.329073422 +0000 UTC m=+2676.896880253" watchObservedRunningTime="2026-11-01 06:54:25.330248699 +0000 UTC m=+2676.898055528"
Nov 01 06:54:26.225202 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.225167    4468 generic.go:356] "Generic (PLEG): container finished" podID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerID="5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc" exitCode=0
Nov 01 06:54:26.225721 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.225242    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" event={"ID":"b3bc3aa9-90af-4e8c-bb37-e614ca82d170","Type":"ContainerDied","Data":"5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"}
Nov 01 06:54:26.225792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.225738    4468 scope.go:122] "RemoveContainer" containerID="5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"
Nov 01 06:54:26.226940 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.226919    4468 generic.go:356] "Generic (PLEG): container finished" podID="71060936-60b8-4df6-bfdb-aa955b031810" containerID="62d6b8057462d932f2f4f81827477f6400252fc3900def4215a0fbcb6af2bcb1" exitCode=0
Nov 01 06:54:26.227033 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.227003    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr" event={"ID":"71060936-60b8-4df6-bfdb-aa955b031810","Type":"ContainerDied","Data":"62d6b8057462d932f2f4f81827477f6400252fc3900def4215a0fbcb6af2bcb1"}
Nov 01 06:54:26.227378 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.227349    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:54:26.227447 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.227435    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:54:26.227514 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:26.227485    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:54:27.235163 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.235127    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" event={"ID":"b3bc3aa9-90af-4e8c-bb37-e614ca82d170","Type":"ContainerStarted","Data":"74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c"}
Nov 01 06:54:27.397027 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.397002    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:27.471984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.471951    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498\" (UniqueName: \"kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498\") pod \"71060936-60b8-4df6-bfdb-aa955b031810\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") "
Nov 01 06:54:27.472191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.472049    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca\") pod \"71060936-60b8-4df6-bfdb-aa955b031810\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") "
Nov 01 06:54:27.472191 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.472149    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca\") pod \"71060936-60b8-4df6-bfdb-aa955b031810\" (UID: \"71060936-60b8-4df6-bfdb-aa955b031810\") "
Nov 01 06:54:27.472495 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.472441    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca" pod "71060936-60b8-4df6-bfdb-aa955b031810" (UID: "71060936-60b8-4df6-bfdb-aa955b031810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:27.472698 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.472680    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:27.474375 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.474352    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca" pod "71060936-60b8-4df6-bfdb-aa955b031810" (UID: "71060936-60b8-4df6-bfdb-aa955b031810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:54:27.474450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.474373    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498" pod "71060936-60b8-4df6-bfdb-aa955b031810" (UID: "71060936-60b8-4df6-bfdb-aa955b031810"). InnerVolumeSpecName "kube-api-access-fv498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:27.574625 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.574591    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71060936-60b8-4df6-bfdb-aa955b031810-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:27.574625 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:27.574625    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv498\" (UniqueName: \"kubernetes.io/projected/71060936-60b8-4df6-bfdb-aa955b031810-kube-api-access-fv498\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:28.232908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:28.232863    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:54:28.234145 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:28.234121    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 01 06:54:28.234220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:28.234183    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 01 06:54:28.244516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:28.244490    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"
Nov 01 06:54:28.244516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:28.244506    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr" event={"ID":"71060936-60b8-4df6-bfdb-aa955b031810","Type":"ContainerDied","Data":"e55e116b448298cee8b6ce762538759a97f25208b9f28d81cd160bdf20a41f54"}
Nov 01 06:54:28.244929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:28.244555    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e55e116b448298cee8b6ce762538759a97f25208b9f28d81cd160bdf20a41f54"
Nov 01 06:54:29.232093 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:29.232057    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 01 06:54:29.232276 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:29.232108    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 01 06:54:29.460126 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:29.460096    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:54:30.233180 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:30.233144    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 01 06:54:30.233358 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:30.233199    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 01 06:54:31.234898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:31.234857    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 01 06:54:31.235330 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:31.234922    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 01 06:54:32.280136 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:32.280104    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:54:32.281988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:32.281959    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 06:54:32.947702 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:54:32.947659    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:54:38.013219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.013185    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"]
Nov 01 06:54:38.013817 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.013803    4468 cpu_manager.go:414] "containerMap: removing container" podUID="71060936-60b8-4df6-bfdb-aa955b031810" containerName="revoke-keys"
Nov 01 06:54:38.013863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.013819    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71060936-60b8-4df6-bfdb-aa955b031810" containerName="revoke-keys"
Nov 01 06:54:38.013939 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.013924    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="71060936-60b8-4df6-bfdb-aa955b031810" containerName="revoke-keys"
Nov 01 06:54:38.016850 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.016823    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.034937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.034911    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrr9\" (UniqueName: \"kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.035090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.034949    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.035174 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.035147    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.119088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.119038    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:54:38.119088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.119074    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:54:38.119088 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.119074    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:54:38.119397 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.119074    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:54:38.122374 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.122353    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"]
Nov 01 06:54:38.136776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.136744    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.136968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.136895    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrr9\" (UniqueName: \"kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.136968 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.136938    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.137656 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.137636    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.139417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.139392    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.239491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.239337    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrr9\" (UniqueName: \"kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9\") pod \"maas-api-revoke-sub-f8abcc32-rk4sf\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.329215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.329181    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:38.702831 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:38.702804    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"]
Nov 01 06:54:38.704701 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:54:38.704672    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2288f9_6685_49bd_a1aa_a7bc6360f3a8.slice/crio-92397f63321641ab5736ae908bb2a1ca4ac79d37b8bdfba4c47bccde9beb6c02 WatchSource:0}: Error finding container 92397f63321641ab5736ae908bb2a1ca4ac79d37b8bdfba4c47bccde9beb6c02: Status 404 returned error can't find the container with id 92397f63321641ab5736ae908bb2a1ca4ac79d37b8bdfba4c47bccde9beb6c02
Nov 01 06:54:39.322864 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:39.322832    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf" event={"ID":"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8","Type":"ContainerStarted","Data":"8627f9bf66c3037926f18dada734f80768197fb04bf01889b7200d9dfc2277af"}
Nov 01 06:54:39.323256 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:39.322881    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf" event={"ID":"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8","Type":"ContainerStarted","Data":"92397f63321641ab5736ae908bb2a1ca4ac79d37b8bdfba4c47bccde9beb6c02"}
Nov 01 06:54:39.436794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:39.436742    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf" podStartSLOduration=2.4367273210000002 podStartE2EDuration="2.436727321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:39.433841518 +0000 UTC m=+2691.001648348" watchObservedRunningTime="2026-11-01 06:54:39.436727321 +0000 UTC m=+2691.004534149"
Nov 01 06:54:40.329981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:40.329950    4468 generic.go:356] "Generic (PLEG): container finished" podID="4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" containerID="8627f9bf66c3037926f18dada734f80768197fb04bf01889b7200d9dfc2277af" exitCode=0
Nov 01 06:54:40.330420 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:40.330023    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf" event={"ID":"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8","Type":"ContainerDied","Data":"8627f9bf66c3037926f18dada734f80768197fb04bf01889b7200d9dfc2277af"}
Nov 01 06:54:41.489804 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.489779    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:41.580860 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.580828    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca\") pod \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") "
Nov 01 06:54:41.581055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.580929    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9\" (UniqueName: \"kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9\") pod \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") "
Nov 01 06:54:41.581055 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.581017    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca\") pod \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\" (UID: \"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8\") "
Nov 01 06:54:41.581241 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.581217    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca" pod "4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" (UID: "4f2288f9-6685-49bd-a1aa-a7bc6360f3a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:41.581395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.581380    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:41.583123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.583100    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca" pod "4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" (UID: "4f2288f9-6685-49bd-a1aa-a7bc6360f3a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:54:41.583123 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.583114    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9" pod "4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" (UID: "4f2288f9-6685-49bd-a1aa-a7bc6360f3a8"). InnerVolumeSpecName "kube-api-access-xlrr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:41.683038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.683004    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrr9\" (UniqueName: \"kubernetes.io/projected/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-kube-api-access-xlrr9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:41.683038 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:41.683035    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:42.344300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:42.344267    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"
Nov 01 06:54:42.344511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:42.344270    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf" event={"ID":"4f2288f9-6685-49bd-a1aa-a7bc6360f3a8","Type":"ContainerDied","Data":"92397f63321641ab5736ae908bb2a1ca4ac79d37b8bdfba4c47bccde9beb6c02"}
Nov 01 06:54:42.344511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:42.344398    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92397f63321641ab5736ae908bb2a1ca4ac79d37b8bdfba4c47bccde9beb6c02"
Nov 01 06:54:46.574501 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.574459    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"]
Nov 01 06:54:46.575108 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.575091    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" containerName="revoke-keys"
Nov 01 06:54:46.575156 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.575110    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" containerName="revoke-keys"
Nov 01 06:54:46.575221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.575212    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" containerName="revoke-keys"
Nov 01 06:54:46.578425 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.578408    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.645290 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.645255    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.645477 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.645374    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4648\" (UniqueName: \"kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.645477 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.645402    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.645573 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.645537    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.645623 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.645611    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.645661 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.645642    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.653685 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.653662    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"]
Nov 01 06:54:46.680842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.680805    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a3e841\"/\"default-dockercfg-846kg\""
Nov 01 06:54:46.681042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.680864    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a3e841\"/\"openshift-service-ca.crt\""
Nov 01 06:54:46.681042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.680868    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a3e841\"/\"autores-model-a3e841-kserve-self-signed-certs\""
Nov 01 06:54:46.681042 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.680874    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a3e841\"/\"kube-root-ca.crt\""
Nov 01 06:54:46.751559 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.751523    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.752435 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.752412    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4648\" (UniqueName: \"kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.752675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.752657    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.753085 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.753062    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.753304 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.753284    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.753828 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.753809    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.754418 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.753104    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.754587 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.753730    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.754681 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.752286    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.757335 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.757313    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.758312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.758289    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.809323 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.809294    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4648\" (UniqueName: \"kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648\") pod \"autores-model-a3e841-kserve-69f6b99c69-d2dkj\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") " pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:46.891450 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:46.891366    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:47.265417 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:47.265376    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"]
Nov 01 06:54:47.266676 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:54:47.266651    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b4091d8_652a_4be9_8932_cd2dbcb3666d.slice/crio-c56d1c6a6c0e97d5d72d5e9ef15137a6620c01929ca50d0b4a5013c0429b8666 WatchSource:0}: Error finding container c56d1c6a6c0e97d5d72d5e9ef15137a6620c01929ca50d0b4a5013c0429b8666: Status 404 returned error can't find the container with id c56d1c6a6c0e97d5d72d5e9ef15137a6620c01929ca50d0b4a5013c0429b8666
Nov 01 06:54:47.382353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:47.382313    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj" event={"ID":"6b4091d8-652a-4be9-8932-cd2dbcb3666d","Type":"ContainerStarted","Data":"1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040"}
Nov 01 06:54:47.382353 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:47.382367    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj" event={"ID":"6b4091d8-652a-4be9-8932-cd2dbcb3666d","Type":"ContainerStarted","Data":"c56d1c6a6c0e97d5d72d5e9ef15137a6620c01929ca50d0b4a5013c0429b8666"}
Nov 01 06:54:47.497237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:47.497182    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj" podStartSLOduration=1.497166901 podStartE2EDuration="1.497166901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:47.495206555 +0000 UTC m=+2699.063013383" watchObservedRunningTime="2026-11-01 06:54:47.497166901 +0000 UTC m=+2699.064973728"
Nov 01 06:54:47.892406 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:47.892369    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:49.942767 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:49.942733    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:54:51.489338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:51.489296    4468 scope.go:122] "RemoveContainer" containerID="cf15d35fae6c463ae6396cc790b7d901f71eafae77c728ef6b1c02456a0c47fe"
Nov 01 06:54:51.512444 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:51.512418    4468 scope.go:122] "RemoveContainer" containerID="d955ca28d1e0d9312f0085ef248e275ecc93f68dec9472b405a6ed209ab08597"
Nov 01 06:54:51.539842 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:51.539670    4468 scope.go:122] "RemoveContainer" containerID="a76edff6d9f8ab22015e7c19d19f45cbb5ab5c1c6110efbdac2c5f2771a2be98"
Nov 01 06:54:51.552162 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:51.552136    4468 scope.go:122] "RemoveContainer" containerID="ac8740368269f237764565253df3790c0020dea32523b9743a9928cc6a210344"
Nov 01 06:54:56.891821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:56.891783    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:56.903955 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:56.903928    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:54:57.614898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:54:57.614861    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:55:02.948455 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:55:02.948417    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:55:17.064184 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.064147    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"]
Nov 01 06:55:17.068122 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.068103    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.144421 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.144374    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"]
Nov 01 06:55:17.171676 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.171647    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-28c598\"/\"premium-w3-28c598-kserve-self-signed-certs\""
Nov 01 06:55:17.230308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.230269    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.230504 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.230330    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzd8\" (UniqueName: \"kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.230578 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.230549    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.230621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.230596    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.230740 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.230724    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.230776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.230764    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332200    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332272    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332378    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332426    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332554    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332605    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzd8\" (UniqueName: \"kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.332862 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.332834    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.333287 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.333217    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.333385 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.333228    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.335317 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.335294    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.335404 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.335297    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.390127 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.390098    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzd8\" (UniqueName: \"kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8\") pod \"premium-w3-28c598-kserve-58bbf6b898-7blks\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") " pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:17.681405 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:17.681313    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:18.059534 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:18.059502    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"]
Nov 01 06:55:18.060544 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:55:18.060514    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34dd1ad4_aed2_42bd_8d09_a01312d1fb84.slice/crio-e7551ea7319bf043e25296bdf68a61907f55b7e70d81231733a5fa994d921c05 WatchSource:0}: Error finding container e7551ea7319bf043e25296bdf68a61907f55b7e70d81231733a5fa994d921c05: Status 404 returned error can't find the container with id e7551ea7319bf043e25296bdf68a61907f55b7e70d81231733a5fa994d921c05
Nov 01 06:55:18.763724 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:18.763691    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks" event={"ID":"34dd1ad4-aed2-42bd-8d09-a01312d1fb84","Type":"ContainerStarted","Data":"ad1ea9ded7ef036724efd22c92a834007cb8978ad63e4d55c6d8bcd78b8a6f93"}
Nov 01 06:55:18.764124 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:18.763734    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks" event={"ID":"34dd1ad4-aed2-42bd-8d09-a01312d1fb84","Type":"ContainerStarted","Data":"e7551ea7319bf043e25296bdf68a61907f55b7e70d81231733a5fa994d921c05"}
Nov 01 06:55:18.877711 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:18.877660    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks" podStartSLOduration=1.8776453640000001 podStartE2EDuration="1.877645364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:18.875250299 +0000 UTC m=+2730.443057127" watchObservedRunningTime="2026-11-01 06:55:18.877645364 +0000 UTC m=+2730.445452192"
Nov 01 06:55:19.681571 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:19.681526    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:21.715546 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:21.715513    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:55:27.681830 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:27.681797    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:27.694518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:27.694486    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:27.828854 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:27.828824    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 06:55:32.946892 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:55:32.946815    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:55:33.195875 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:33.195834    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"]
Nov 01 06:55:36.915599 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:36.915556    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"]
Nov 01 06:55:38.196620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.196580    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4" podUID="10fa6e73-5678-4f31-b603-71204e8e9ea2" containerName="payload-processing" containerID="cri-o://d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93" gracePeriod=25
Nov 01 06:55:38.467170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.467145    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:55:38.538659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.538636    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r\") pod \"10fa6e73-5678-4f31-b603-71204e8e9ea2\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") "
Nov 01 06:55:38.538790 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.538720    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume\") pod \"10fa6e73-5678-4f31-b603-71204e8e9ea2\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") "
Nov 01 06:55:38.538832 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.538810    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials\") pod \"10fa6e73-5678-4f31-b603-71204e8e9ea2\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") "
Nov 01 06:55:38.538904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.538890    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume\") pod \"10fa6e73-5678-4f31-b603-71204e8e9ea2\" (UID: \"10fa6e73-5678-4f31-b603-71204e8e9ea2\") "
Nov 01 06:55:38.539070 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.539040    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume" pod "10fa6e73-5678-4f31-b603-71204e8e9ea2" (UID: "10fa6e73-5678-4f31-b603-71204e8e9ea2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:55:38.539167 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.539148    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume" pod "10fa6e73-5678-4f31-b603-71204e8e9ea2" (UID: "10fa6e73-5678-4f31-b603-71204e8e9ea2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:55:38.539395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.539373    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:38.539395 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.539398    4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10fa6e73-5678-4f31-b603-71204e8e9ea2-routing-overlay-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:38.540972 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.540951    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials" pod "10fa6e73-5678-4f31-b603-71204e8e9ea2" (UID: "10fa6e73-5678-4f31-b603-71204e8e9ea2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:55:38.541057 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.541027    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r" pod "10fa6e73-5678-4f31-b603-71204e8e9ea2" (UID: "10fa6e73-5678-4f31-b603-71204e8e9ea2"). InnerVolumeSpecName "kube-api-access-wxt7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:55:38.641028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.641004    4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-provider-credentials\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:38.641028 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.641028    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxt7r\" (UniqueName: \"kubernetes.io/projected/10fa6e73-5678-4f31-b603-71204e8e9ea2-kube-api-access-wxt7r\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:38.892608 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.892576    4468 generic.go:356] "Generic (PLEG): container finished" podID="10fa6e73-5678-4f31-b603-71204e8e9ea2" containerID="d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93" exitCode=0
Nov 01 06:55:38.892783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.892636    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4" event={"ID":"10fa6e73-5678-4f31-b603-71204e8e9ea2","Type":"ContainerDied","Data":"d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93"}
Nov 01 06:55:38.892783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.892647    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"
Nov 01 06:55:38.892783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.892679    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4" event={"ID":"10fa6e73-5678-4f31-b603-71204e8e9ea2","Type":"ContainerDied","Data":"bcd50a858a5553dc10ea299ced9528e2f82d61bf115b1a54ccbf0251e175ac4e"}
Nov 01 06:55:38.892783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.892702    4468 scope.go:122] "RemoveContainer" containerID="d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93"
Nov 01 06:55:38.902511 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.902459    4468 scope.go:122] "RemoveContainer" containerID="d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93"
Nov 01 06:55:38.902768 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:55:38.902745    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93\": container with ID starting with d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93 not found: ID does not exist" containerID="d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93"
Nov 01 06:55:38.902846 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:38.902780    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93"} err="failed to get container status \"d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93\": rpc error: code = NotFound desc = could not find container \"d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93\": container with ID starting with d04af8580bb2882988aa5b075c3921a7dbf4a957df89ac4016b2f4550dc05d93 not found: ID does not exist"
Nov 01 06:55:39.059730 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:39.059704    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"]
Nov 01 06:55:39.109090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:39.109062    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd7sn5l4"]
Nov 01 06:55:40.590215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:40.590182    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fa6e73-5678-4f31-b603-71204e8e9ea2" path="/var/lib/kubelet/pods/10fa6e73-5678-4f31-b603-71204e8e9ea2/volumes"
Nov 01 06:55:51.935933 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:51.935837    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj" podUID="6b4091d8-652a-4be9-8932-cd2dbcb3666d" containerName="main" containerID="cri-o://1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040" gracePeriod=45
Nov 01 06:55:52.203300 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.203226    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:55:52.255509 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255448    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home\") pod \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") "
Nov 01 06:55:52.255683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255564    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache\") pod \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") "
Nov 01 06:55:52.255683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255587    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648\" (UniqueName: \"kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648\") pod \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") "
Nov 01 06:55:52.255683 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255659    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm\") pod \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") "
Nov 01 06:55:52.255827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home" pod "6b4091d8-652a-4be9-8932-cd2dbcb3666d" (UID: "6b4091d8-652a-4be9-8932-cd2dbcb3666d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:52.255827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255743    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir\") pod \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") "
Nov 01 06:55:52.255827 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255785    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache" pod "6b4091d8-652a-4be9-8932-cd2dbcb3666d" (UID: "6b4091d8-652a-4be9-8932-cd2dbcb3666d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:52.255922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255880    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs\") pod \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\" (UID: \"6b4091d8-652a-4be9-8932-cd2dbcb3666d\") "
Nov 01 06:55:52.255965 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.255944    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir" pod "6b4091d8-652a-4be9-8932-cd2dbcb3666d" (UID: "6b4091d8-652a-4be9-8932-cd2dbcb3666d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:52.256428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.256351    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:52.256428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.256371    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:52.256428 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.256380    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:52.257966 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.257938    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm" pod "6b4091d8-652a-4be9-8932-cd2dbcb3666d" (UID: "6b4091d8-652a-4be9-8932-cd2dbcb3666d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:55:52.258075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.258041    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs" pod "6b4091d8-652a-4be9-8932-cd2dbcb3666d" (UID: "6b4091d8-652a-4be9-8932-cd2dbcb3666d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:55:52.258075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.258051    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648" pod "6b4091d8-652a-4be9-8932-cd2dbcb3666d" (UID: "6b4091d8-652a-4be9-8932-cd2dbcb3666d"). InnerVolumeSpecName "kube-api-access-c4648". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:55:52.357138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.357104    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4091d8-652a-4be9-8932-cd2dbcb3666d-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:52.357138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.357136    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4648\" (UniqueName: \"kubernetes.io/projected/6b4091d8-652a-4be9-8932-cd2dbcb3666d-kube-api-access-c4648\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:52.357354 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:52.357150    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4091d8-652a-4be9-8932-cd2dbcb3666d-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:55:53.013794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.013712    4468 generic.go:356] "Generic (PLEG): container finished" podID="6b4091d8-652a-4be9-8932-cd2dbcb3666d" containerID="1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040" exitCode=0
Nov 01 06:55:53.013794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.013759    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"
Nov 01 06:55:53.014361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.013793    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj" event={"ID":"6b4091d8-652a-4be9-8932-cd2dbcb3666d","Type":"ContainerDied","Data":"1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040"}
Nov 01 06:55:53.014361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.013843    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj" event={"ID":"6b4091d8-652a-4be9-8932-cd2dbcb3666d","Type":"ContainerDied","Data":"c56d1c6a6c0e97d5d72d5e9ef15137a6620c01929ca50d0b4a5013c0429b8666"}
Nov 01 06:55:53.014361 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.013862    4468 scope.go:122] "RemoveContainer" containerID="1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040"
Nov 01 06:55:53.023280 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.023260    4468 scope.go:122] "RemoveContainer" containerID="1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040"
Nov 01 06:55:53.023543 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:55:53.023524    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040\": container with ID starting with 1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040 not found: ID does not exist" containerID="1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040"
Nov 01 06:55:53.023607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.023551    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040"} err="failed to get container status \"1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040\": rpc error: code = NotFound desc = could not find container \"1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040\": container with ID starting with 1af321c123a5f88abceede528d7d8bfe380f50324d9e9348b70b7d4247ec2040 not found: ID does not exist"
Nov 01 06:55:53.177138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.177107    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"]
Nov 01 06:55:53.226584 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:53.226551    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a3e841/autores-model-a3e841-kserve-69f6b99c69-d2dkj"]
Nov 01 06:55:54.590678 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:54.590642    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4091d8-652a-4be9-8932-cd2dbcb3666d" path="/var/lib/kubelet/pods/6b4091d8-652a-4be9-8932-cd2dbcb3666d/volumes"
Nov 01 06:55:56.050344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:56.050309    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"]
Nov 01 06:55:56.099981 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:56.099939    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a8702c6-lvdwd"]
Nov 01 06:55:56.590121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:55:56.590089    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1b84c7-e06a-45cb-80e8-d42acff6e581" path="/var/lib/kubelet/pods/8a1b84c7-e06a-45cb-80e8-d42acff6e581/volumes"
Nov 01 06:56:02.947805 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:56:02.947763    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:56:05.366932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.366895    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"]
Nov 01 06:56:05.368059 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.368025    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6b4091d8-652a-4be9-8932-cd2dbcb3666d" containerName="main"
Nov 01 06:56:05.368059 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.368049    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4091d8-652a-4be9-8932-cd2dbcb3666d" containerName="main"
Nov 01 06:56:05.368254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.368168    4468 cpu_manager.go:414] "containerMap: removing container" podUID="10fa6e73-5678-4f31-b603-71204e8e9ea2" containerName="payload-processing"
Nov 01 06:56:05.368254 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.368178    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fa6e73-5678-4f31-b603-71204e8e9ea2" containerName="payload-processing"
Nov 01 06:56:05.368497 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.368451    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fa6e73-5678-4f31-b603-71204e8e9ea2" containerName="payload-processing"
Nov 01 06:56:05.368582 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.368499    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4091d8-652a-4be9-8932-cd2dbcb3666d" containerName="main"
Nov 01 06:56:05.373735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.373711    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.425018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.424981    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"]
Nov 01 06:56:05.470794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.470753    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 06:56:05.470794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.470773    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:56:05.471006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.470841    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 06:56:05.474886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.474868    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 06:56:05.523973 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.523937    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.524138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.524053    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.524138 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.524086    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdvf\" (UniqueName: \"kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.625607 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.625521    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.625770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.625623    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.625770 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.625652    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdvf\" (UniqueName: \"kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.626307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.626278    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.628031 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.628011    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.681726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.681691    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdvf\" (UniqueName: \"kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf\") pod \"maas-api-revoke-sub-665e906c-bhkns\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:05.687477 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:05.687442    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:06.065932 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:06.065907    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"]
Nov 01 06:56:06.068087 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:56:06.068060    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b149084_f7e3_4d91_8b29_945ecc8dbb68.slice/crio-8ad8048910a6816a95bea5d482747da921859394adfd4992cd65d363f9cffb9c WatchSource:0}: Error finding container 8ad8048910a6816a95bea5d482747da921859394adfd4992cd65d363f9cffb9c: Status 404 returned error can't find the container with id 8ad8048910a6816a95bea5d482747da921859394adfd4992cd65d363f9cffb9c
Nov 01 06:56:06.098821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:06.098797    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns" event={"ID":"3b149084-f7e3-4d91-8b29-945ecc8dbb68","Type":"ContainerStarted","Data":"8ad8048910a6816a95bea5d482747da921859394adfd4992cd65d363f9cffb9c"}
Nov 01 06:56:07.106453 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:07.106418    4468 generic.go:356] "Generic (PLEG): container finished" podID="3b149084-f7e3-4d91-8b29-945ecc8dbb68" containerID="60d8e96fe5efcb2fe8246301b4e449faf057d9d22bcea05aa6ccda71912ba83e" exitCode=0
Nov 01 06:56:07.106867 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:07.106514    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns" event={"ID":"3b149084-f7e3-4d91-8b29-945ecc8dbb68","Type":"ContainerDied","Data":"60d8e96fe5efcb2fe8246301b4e449faf057d9d22bcea05aa6ccda71912ba83e"}
Nov 01 06:56:08.253443 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.253416    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:08.363847 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.363780    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf\" (UniqueName: \"kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf\") pod \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") "
Nov 01 06:56:08.364002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.363866    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca\") pod \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") "
Nov 01 06:56:08.364002 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.363981    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca\") pod \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\" (UID: \"3b149084-f7e3-4d91-8b29-945ecc8dbb68\") "
Nov 01 06:56:08.364301 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.364272    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca" pod "3b149084-f7e3-4d91-8b29-945ecc8dbb68" (UID: "3b149084-f7e3-4d91-8b29-945ecc8dbb68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:56:08.364542 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.364522    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:08.366551 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.366515    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca" pod "3b149084-f7e3-4d91-8b29-945ecc8dbb68" (UID: "3b149084-f7e3-4d91-8b29-945ecc8dbb68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:56:08.366668 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.366597    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf" pod "3b149084-f7e3-4d91-8b29-945ecc8dbb68" (UID: "3b149084-f7e3-4d91-8b29-945ecc8dbb68"). InnerVolumeSpecName "kube-api-access-2pdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:56:08.465926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.465894    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdvf\" (UniqueName: \"kubernetes.io/projected/3b149084-f7e3-4d91-8b29-945ecc8dbb68-kube-api-access-2pdvf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:08.465926 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:08.465920    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b149084-f7e3-4d91-8b29-945ecc8dbb68-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:09.119882 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:09.119850    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns" event={"ID":"3b149084-f7e3-4d91-8b29-945ecc8dbb68","Type":"ContainerDied","Data":"8ad8048910a6816a95bea5d482747da921859394adfd4992cd65d363f9cffb9c"}
Nov 01 06:56:09.120061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:09.119896    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad8048910a6816a95bea5d482747da921859394adfd4992cd65d363f9cffb9c"
Nov 01 06:56:09.120061 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:09.119907    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"
Nov 01 06:56:09.882702 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:09.882669    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:56:19.504186 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.504154    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"]
Nov 01 06:56:19.504732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.504719    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3b149084-f7e3-4d91-8b29-945ecc8dbb68" containerName="revoke-keys"
Nov 01 06:56:19.504776 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.504734    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b149084-f7e3-4d91-8b29-945ecc8dbb68" containerName="revoke-keys"
Nov 01 06:56:19.504823 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.504814    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b149084-f7e3-4d91-8b29-945ecc8dbb68" containerName="revoke-keys"
Nov 01 06:56:19.508018 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.508003    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.582924 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.582894    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"]
Nov 01 06:56:19.593731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.593705    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.593941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.593742    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.593941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.593832    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpp6\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.593941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.593878    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.593941 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.593894    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.594183 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.594051    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.594183 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.594134    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.594267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.594189    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.594267 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.594249    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.609597 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.609573    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-3db672-openshift-default-dockercfg-dng52\""
Nov 01 06:56:19.695215 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695176    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695238    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695279    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpp6\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695326    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695343    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695371    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695408    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695446    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695484    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695748 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695671    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695912 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695893    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695893    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.695983 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.695918    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.696074 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.696050    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.697922 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.697898    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.698239 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.698219    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.753208 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.753172    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.753351 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.753284    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpp6\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:19.821590 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:19.821558    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:20.251855 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:20.251818    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"]
Nov 01 06:56:20.259784 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:56:20.259747    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3e1cbd_bfea_4a36_8e99_52507f1c8b23.slice/crio-7e17f5f79e393bffb85453cde8dbf75383654c17f6075b88f583ad5499f45cc9 WatchSource:0}: Error finding container 7e17f5f79e393bffb85453cde8dbf75383654c17f6075b88f583ad5499f45cc9: Status 404 returned error can't find the container with id 7e17f5f79e393bffb85453cde8dbf75383654c17f6075b88f583ad5499f45cc9
Nov 01 06:56:20.262531 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:20.262484    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:20.262640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:20.262579    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:20.262640 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:20.262623    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:21.194870 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:21.194830    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" event={"ID":"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23","Type":"ContainerStarted","Data":"3c46bdace1549578bb13ebf9ae0188c1ad7d37eae0670a8440e1b1e19f6d13b9"}
Nov 01 06:56:21.195289 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:21.194898    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" event={"ID":"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23","Type":"ContainerStarted","Data":"7e17f5f79e393bffb85453cde8dbf75383654c17f6075b88f583ad5499f45cc9"}
Nov 01 06:56:21.314409 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:21.314354    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podStartSLOduration=2.314339446 podStartE2EDuration="2.314339446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:21.31076908 +0000 UTC m=+2792.878575910" watchObservedRunningTime="2026-11-01 06:56:21.314339446 +0000 UTC m=+2792.882146274"
Nov 01 06:56:21.826783 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:21.826699    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:21.866062 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:21.866029    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:22.202281 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:22.202190    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:22.203232 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:22.203205    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:56:32.948319 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:56:32.948282    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:56:41.100185 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.098506    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"]
Nov 01 06:56:41.105016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.104987    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.110937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.110911    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.110942    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111051 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.110964    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111139 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.111123    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111187 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.111178    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111291 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.111274    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111401 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.111377    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.111434    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhnb\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.111554 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.111454    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.243631 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.243579    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.245441 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.245414    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhnb\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.246065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.245182    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.246240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.246034    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.268502 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.262091    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.272988 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.272957    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.273863 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.273831    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.274861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.274836    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.275861 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.275743    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.276384 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.276367    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.276942 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.269071    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.277092 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.269778    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c831a4c0-openshift-default-dockercfg-mtmts\""
Nov 01 06:56:41.277197 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.276259    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.277307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.274672    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.277416 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.276773    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.295888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.295847    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.296794 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.296758    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.305929 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.297608    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"]
Nov 01 06:56:41.328659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.327637    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhnb\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.334513 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.332165    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.420845 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.420742    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:41.848726 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.848695    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"]
Nov 01 06:56:41.887550 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:56:41.887522    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f86236_8114_49f0_bb7c_fa399fde0574.slice/crio-12cf2681001ac6cb784689d64b10690626578d9f7e9df59735d5bf68fa326c58 WatchSource:0}: Error finding container 12cf2681001ac6cb784689d64b10690626578d9f7e9df59735d5bf68fa326c58: Status 404 returned error can't find the container with id 12cf2681001ac6cb784689d64b10690626578d9f7e9df59735d5bf68fa326c58
Nov 01 06:56:41.890093 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.890056    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:41.890198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.890142    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:41.890198 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:41.890187    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:42.363303 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:42.363253    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" event={"ID":"44f86236-8114-49f0-bb7c-fa399fde0574","Type":"ContainerStarted","Data":"0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"}
Nov 01 06:56:42.363821 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:42.363333    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" event={"ID":"44f86236-8114-49f0-bb7c-fa399fde0574","Type":"ContainerStarted","Data":"12cf2681001ac6cb784689d64b10690626578d9f7e9df59735d5bf68fa326c58"}
Nov 01 06:56:42.421705 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:42.421678    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:42.427437 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:42.427415    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:42.484231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:42.484180    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podStartSLOduration=1.484166867 podStartE2EDuration="1.484166867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:42.479811979 +0000 UTC m=+2814.047618808" watchObservedRunningTime="2026-11-01 06:56:42.484166867 +0000 UTC m=+2814.051973695"
Nov 01 06:56:43.369170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:43.369138    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:43.370010 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:43.369990    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:56:50.318691 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:50.318644    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"]
Nov 01 06:56:50.319177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:50.318922    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" containerID="cri-o://fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd" gracePeriod=30
Nov 01 06:56:51.427984 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.427944    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"]
Nov 01 06:56:51.432170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.432149    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.592524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.586919    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0cb4ef-openshift-default-dockercfg-dvgr6\""
Nov 01 06:56:51.592524 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.592296    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"]
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596137    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkcm\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596183    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596249    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596321    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596399    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596509    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596640    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596676    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.597016 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.596715    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697328    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697393    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697414    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697435    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697505    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkcm\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697525    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697564    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697626    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.697701 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697691    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.698090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697741    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.698090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.697845    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.698090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.698006    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.698090 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.698033    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.698508 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.698153    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.700065 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.700036    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.700240 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.700220    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.754660 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.754627    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:51.756170 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:51.756148    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkcm\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm\") pod \"e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") " pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:52.046369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.046332    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:52.414107 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.414018    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:56:52.528877 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.528847    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"]
Nov 01 06:56:52.529782 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:56:52.529756    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be60b38_5f66_4a1b_9891_af1ec26fcd24.slice/crio-862f4e5a6211a5508af0fac147b5e92eb8f121c748cc5762e7333835af6087be WatchSource:0}: Error finding container 862f4e5a6211a5508af0fac147b5e92eb8f121c748cc5762e7333835af6087be: Status 404 returned error can't find the container with id 862f4e5a6211a5508af0fac147b5e92eb8f121c748cc5762e7333835af6087be
Nov 01 06:56:52.532682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.532647    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:52.532808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.532726    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:52.532808 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.532772    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:56:52.655134 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.655100    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" event={"ID":"1be60b38-5f66-4a1b-9891-af1ec26fcd24","Type":"ContainerStarted","Data":"9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"}
Nov 01 06:56:52.655305 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.655153    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" event={"ID":"1be60b38-5f66-4a1b-9891-af1ec26fcd24","Type":"ContainerStarted","Data":"862f4e5a6211a5508af0fac147b5e92eb8f121c748cc5762e7333835af6087be"}
Nov 01 06:56:52.772006 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:52.771404    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podStartSLOduration=1.7713853689999999 podStartE2EDuration="1.771385369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:52.771087444 +0000 UTC m=+2824.338894274" watchObservedRunningTime="2026-11-01 06:56:52.771385369 +0000 UTC m=+2824.339192197"
Nov 01 06:56:53.047369 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:53.047344    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:53.052837 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:53.052814    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:53.670779 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:53.670742    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:53.671731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:53.671710    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:56:54.962921 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:54.962881    4468 patch_prober.go:29] interesting pod/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:56:54.963307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:54.962942    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:56:55.595520 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.595447    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:56:55.649904 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.649866    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.649940    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.649971    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650006    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650034    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650111 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650085    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650245    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650278    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650311    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:56:55.650370 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650350    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:56:55.650642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650381    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket\") pod \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\" (UID: \"ae9751b1-0e80-47d2-a676-91cb0edd19fa\") "
Nov 01 06:56:55.650642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650605    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:56:55.650735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650687    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:56:55.650970 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.650944    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:56:55.651408 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.651382    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.651516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.651414    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.651516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.651429    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.651516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.651442    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.651516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.651456    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.652363 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.652344    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:56:55.652448 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.652412    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:56:55.652745 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.652724    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:56:55.653229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.653213    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc" pod "ae9751b1-0e80-47d2-a676-91cb0edd19fa" (UID: "ae9751b1-0e80-47d2-a676-91cb0edd19fa"). InnerVolumeSpecName "kube-api-access-l7rtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:56:55.685050 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.685015    4468 generic.go:356] "Generic (PLEG): container finished" podID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerID="fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd" exitCode=0
Nov 01 06:56:55.685229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.685073    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"
Nov 01 06:56:55.685229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.685097    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" event={"ID":"ae9751b1-0e80-47d2-a676-91cb0edd19fa","Type":"ContainerDied","Data":"fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd"}
Nov 01 06:56:55.685229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.685149    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v" event={"ID":"ae9751b1-0e80-47d2-a676-91cb0edd19fa","Type":"ContainerDied","Data":"5aae717655111bf00f251037bb12b901de7f0d137486a41af3bd0bb7b1c88393"}
Nov 01 06:56:55.685229 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.685170    4468 scope.go:122] "RemoveContainer" containerID="fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd"
Nov 01 06:56:55.696046 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.696022    4468 scope.go:122] "RemoveContainer" containerID="d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"
Nov 01 06:56:55.705324 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.705306    4468 scope.go:122] "RemoveContainer" containerID="fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd"
Nov 01 06:56:55.705589 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:56:55.705569    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd\": container with ID starting with fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd not found: ID does not exist" containerID="fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd"
Nov 01 06:56:55.705672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.705601    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd"} err="failed to get container status \"fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd\": rpc error: code = NotFound desc = could not find container \"fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd\": container with ID starting with fd501aac0c852c799266d7a577d2ebe8ec239c8620f9f698355d62c4a60d09fd not found: ID does not exist"
Nov 01 06:56:55.705672 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.705628    4468 scope.go:122] "RemoveContainer" containerID="d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"
Nov 01 06:56:55.705891 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:56:55.705873    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f\": container with ID starting with d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f not found: ID does not exist" containerID="d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"
Nov 01 06:56:55.705938 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.705900    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f"} err="failed to get container status \"d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f\": rpc error: code = NotFound desc = could not find container \"d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f\": container with ID starting with d7be2beadcba136284e7c8677508848a100490a6c358dffb319b2f01a719673f not found: ID does not exist"
Nov 01 06:56:55.753338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.753295    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rtc\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-kube-api-access-l7rtc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.753338 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.753335    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.753583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.753351    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.753583 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.753367    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae9751b1-0e80-47d2-a676-91cb0edd19fa-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:55.876462 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.876383    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"]
Nov 01 06:56:55.882869 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:55.882829    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a3e841-openshift-default-6989c98fc5-q5s9v"]
Nov 01 06:56:56.590833 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:56:56.590797    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" path="/var/lib/kubelet/pods/ae9751b1-0e80-47d2-a676-91cb0edd19fa/volumes"
Nov 01 06:57:02.952483 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:57:02.952388    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:57:17.076613 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:57:17.076575    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:57:32.949399 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:57:32.949362    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:57:51.671924 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:57:51.671884    4468 scope.go:122] "RemoveContainer" containerID="c1eb43e5a5f1f4c2591430db36fbc1ebfd280759874e795ad72ae035f405bc2c"
Nov 01 06:58:01.592360 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:01.592323    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:58:02.954591 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:58:02.954551    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:58:11.186056 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:11.185975    4468 generic.go:356] "Generic (PLEG): container finished" podID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerID="3c46bdace1549578bb13ebf9ae0188c1ad7d37eae0670a8440e1b1e19f6d13b9" exitCode=0
Nov 01 06:58:11.186605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:11.186048    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" event={"ID":"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23","Type":"ContainerDied","Data":"3c46bdace1549578bb13ebf9ae0188c1ad7d37eae0670a8440e1b1e19f6d13b9"}
Nov 01 06:58:11.186605 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:11.186564    4468 scope.go:122] "RemoveContainer" containerID="3c46bdace1549578bb13ebf9ae0188c1ad7d37eae0670a8440e1b1e19f6d13b9"
Nov 01 06:58:11.188362 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:11.188330    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:11.188442 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:11.188420    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:11.188494 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:11.188461    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:12.232066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:12.232032    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" event={"ID":"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23","Type":"ContainerStarted","Data":"73ae624fec91da445608d3507fba0b690cdab660efb07f5ef8273751505b2a86"}
Nov 01 06:58:12.824806 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:12.824772    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:58:12.825177 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:12.825095    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 06:58:12.825348 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:12.825324    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 06:58:13.822498 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:13.822442    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 06:58:13.823133 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:13.823100    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 06:58:14.828598 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:14.828561    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 06:58:14.829231 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:14.829198    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 06:58:15.823914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:15.822517    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 06:58:15.823914 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:15.822594    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 06:58:16.827632 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:16.827596    4468 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 01 06:58:16.828222 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:16.828186    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 01 06:58:17.826732 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:17.826697    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:58:17.827150 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:17.827124    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:58:17.827998 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:17.827978    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 06:58:32.979255 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:58:32.979166    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:58:39.712630 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:39.712591    4468 patch_prober.go:29] interesting pod/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:58:39.713101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:39.712658    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:58:40.432344 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.432315    4468 generic.go:356] "Generic (PLEG): container finished" podID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerID="9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d" exitCode=0
Nov 01 06:58:40.432620 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.432393    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" event={"ID":"1be60b38-5f66-4a1b-9891-af1ec26fcd24","Type":"ContainerDied","Data":"9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"}
Nov 01 06:58:40.432886 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.432867    4468 scope.go:122] "RemoveContainer" containerID="9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"
Nov 01 06:58:40.434169 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.434144    4468 generic.go:356] "Generic (PLEG): container finished" podID="44f86236-8114-49f0-bb7c-fa399fde0574" containerID="0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560" exitCode=0
Nov 01 06:58:40.434262 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.434217    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" event={"ID":"44f86236-8114-49f0-bb7c-fa399fde0574","Type":"ContainerDied","Data":"0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"}
Nov 01 06:58:40.434491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.434440    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:40.434588 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.434572    4468 scope.go:122] "RemoveContainer" containerID="0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"
Nov 01 06:58:40.434663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.434578    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:40.434706 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.434686    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:40.435887 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.435863    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:40.435948 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.435927    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:40.436019 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:40.435964    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:58:41.458849 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:41.458813    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" event={"ID":"1be60b38-5f66-4a1b-9891-af1ec26fcd24","Type":"ContainerStarted","Data":"bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5"}
Nov 01 06:58:41.461081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:41.461057    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" event={"ID":"44f86236-8114-49f0-bb7c-fa399fde0574","Type":"ContainerStarted","Data":"6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f"}
Nov 01 06:58:42.049493 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:42.048644    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:58:42.049710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:42.049553    4468 patch_prober.go:29] interesting pod/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 06:58:42.049710 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:42.049603    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 06:58:42.438490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:42.422965    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:58:42.438490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:42.427330    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 06:58:42.438490 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:42.427385    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 06:58:43.050489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:43.049923    4468 patch_prober.go:29] interesting pod/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 06:58:43.050489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:43.049984    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 06:58:43.369775 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:43.369687    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:58:43.427491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:43.423911    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 06:58:43.427491 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:43.423970    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 06:58:44.058488 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:44.057074    4468 patch_prober.go:29] interesting pod/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 06:58:44.058488 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:44.057146    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 06:58:44.426971 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:44.426871    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 06:58:44.427237 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:44.427207    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 06:58:45.052518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:45.050560    4468 patch_prober.go:29] interesting pod/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 06:58:45.052518 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:45.050617    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 06:58:45.422312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:45.421833    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 06:58:45.422312 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:45.421892    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 06:58:46.048954 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:46.048917    4468 patch_prober.go:29] interesting pod/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 06:58:46.049270 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:46.049228    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 06:58:46.422307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:46.422203    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 01 06:58:46.422307 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:46.422271    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 01 06:58:47.051146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:47.051117    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:58:47.051600 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:47.051579    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:58:47.052176 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:47.052156    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 06:58:47.426643 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:47.426555    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:58:47.427727 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:47.427706    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 06:58:56.490460 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:58:56.490425    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:59:02.949936 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:59:02.949897    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:59:13.649771 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.649737    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"]
Nov 01 06:59:13.650657 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.650638    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy"
Nov 01 06:59:13.650780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.650660    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy"
Nov 01 06:59:13.650780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.650687    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy"
Nov 01 06:59:13.650780 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.650693    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy"
Nov 01 06:59:13.650908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.650869    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy"
Nov 01 06:59:13.650908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.650880    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9751b1-0e80-47d2-a676-91cb0edd19fa" containerName="istio-proxy"
Nov 01 06:59:13.654503 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.654454    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.714489 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.707602    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c52bfa-openshift-default-dockercfg-qwprw\""
Nov 01 06:59:13.755101 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.755055    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.755449 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.755416    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.755763 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.755743    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgm8\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.755949 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.755933    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.756159 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.756134    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.756407 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.756375    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.756731 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.756714    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.760898 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.760868    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.764621 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.764596    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.764956 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.764433    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"]
Nov 01 06:59:13.882481 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.882435    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.883532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.883503    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.883713 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.883550    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.883937 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.883917    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgm8\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.884075 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884059    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.884552 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884533    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.884792 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884740    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.884946 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884924    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.885072 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884420    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.885146 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884925    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.885249 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.885239    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.885667 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.885654    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.886119 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.885563    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.886220 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.884764    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.887424 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.887382    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.889944 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.889922    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.940005 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.939954    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgm8\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.940414 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.940393    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token\") pod \"e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") " pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:13.967735 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:13.967708    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:14.394682 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.394652    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"]
Nov 01 06:59:14.395579 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:59:14.395543    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24aeff4_8c1e_4638_ae19_8c09010af0f6.slice/crio-4c2ea8e0ae056e2ba0a067a1a8ae4e6666a252aadd31df2fc718935f5b672685 WatchSource:0}: Error finding container 4c2ea8e0ae056e2ba0a067a1a8ae4e6666a252aadd31df2fc718935f5b672685: Status 404 returned error can't find the container with id 4c2ea8e0ae056e2ba0a067a1a8ae4e6666a252aadd31df2fc718935f5b672685
Nov 01 06:59:14.397928 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.397894    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:59:14.398015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.397960    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:59:14.398015 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.397986    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 06:59:14.662675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.662586    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" event={"ID":"b24aeff4-8c1e-4638-ae19-8c09010af0f6","Type":"ContainerStarted","Data":"8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"}
Nov 01 06:59:14.662675 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.662638    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" event={"ID":"b24aeff4-8c1e-4638-ae19-8c09010af0f6","Type":"ContainerStarted","Data":"4c2ea8e0ae056e2ba0a067a1a8ae4e6666a252aadd31df2fc718935f5b672685"}
Nov 01 06:59:14.779659 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.779608    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podStartSLOduration=1.779594349 podStartE2EDuration="1.779594349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:14.776645663 +0000 UTC m=+2966.344452502" watchObservedRunningTime="2026-11-01 06:59:14.779594349 +0000 UTC m=+2966.347401248"
Nov 01 06:59:14.968530 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:14.968436    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:15.974089 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:15.974050    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:16.683602 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:16.683571    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:16.684540 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:16.684518    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 06:59:28.046081 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:28.046043    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"]
Nov 01 06:59:28.095664 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:28.095627    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4f828d-rs2fr"]
Nov 01 06:59:28.591642 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:28.591608    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71060936-60b8-4df6-bfdb-aa955b031810" path="/var/lib/kubelet/pods/71060936-60b8-4df6-bfdb-aa955b031810/volumes"
Nov 01 06:59:30.850066 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:30.850034    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 06:59:32.948166 ip-10-0-138-21 kubenswrapper[4468]: E1101 06:59:32.948128    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 06:59:37.703532 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.703496    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"]
Nov 01 06:59:37.707782 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.707764    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.760331 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.760291    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c831a4c0\"/\"kube-root-ca.crt\""
Nov 01 06:59:37.809908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.809861    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c831a4c0\"/\"default-dockercfg-fhc2d\""
Nov 01 06:59:37.809908 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.809886    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c831a4c0\"/\"openshift-service-ca.crt\""
Nov 01 06:59:37.810121 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.809919    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c831a4c0\"/\"e2e-lifecycle-model-c831a4c0-kserve-self-signed-certs\""
Nov 01 06:59:37.813349 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.813330    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"]
Nov 01 06:59:37.831221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.831183    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.831221 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.831231    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.831562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.831393    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.831562 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.831447    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkq7\" (UniqueName: \"kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.831692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.831566    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.831692 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.831592    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933216    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933250 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933250    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933303    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933336    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkq7\" (UniqueName: \"kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933399    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933516 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933419    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933815 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933794    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933818    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.933888 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.933853    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.935788 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.935770    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.936082 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.936065    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:37.989880 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:37.989805    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkq7\" (UniqueName: \"kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7\") pod \"e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") " pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:38.020553 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:38.020518    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:38.395663 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:38.395631    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"]
Nov 01 06:59:38.396824 ip-10-0-138-21 kubenswrapper[4468]: W1101 06:59:38.396794    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15d82ece_9bde_4040_b7f9_aa0c891bc89c.slice/crio-37cac6c9ac51d2a9d4cf26ca6bfd9a8380da325e23af53a8a63a4586802aa76e WatchSource:0}: Error finding container 37cac6c9ac51d2a9d4cf26ca6bfd9a8380da325e23af53a8a63a4586802aa76e: Status 404 returned error can't find the container with id 37cac6c9ac51d2a9d4cf26ca6bfd9a8380da325e23af53a8a63a4586802aa76e
Nov 01 06:59:38.832179 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:38.832144    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd" event={"ID":"15d82ece-9bde-4040-b7f9-aa0c891bc89c","Type":"ContainerStarted","Data":"db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587"}
Nov 01 06:59:38.832637 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:38.832188    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd" event={"ID":"15d82ece-9bde-4040-b7f9-aa0c891bc89c","Type":"ContainerStarted","Data":"37cac6c9ac51d2a9d4cf26ca6bfd9a8380da325e23af53a8a63a4586802aa76e"}
Nov 01 06:59:38.947574 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:38.947519    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd" podStartSLOduration=1.947504102 podStartE2EDuration="1.947504102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:38.944371563 +0000 UTC m=+2990.512178391" watchObservedRunningTime="2026-11-01 06:59:38.947504102 +0000 UTC m=+2990.515310930"
Nov 01 06:59:39.021219 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:39.021190    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:42.045899 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:42.045867    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"]
Nov 01 06:59:42.095286 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:42.095251    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abcc32-rk4sf"]
Nov 01 06:59:42.591135 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:42.591105    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2288f9-6685-49bd-a1aa-a7bc6360f3a8" path="/var/lib/kubelet/pods/4f2288f9-6685-49bd-a1aa-a7bc6360f3a8/volumes"
Nov 01 06:59:48.021308 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:48.021268    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:48.039110 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:48.039078    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 06:59:48.910248 ip-10-0-138-21 kubenswrapper[4468]: I1101 06:59:48.910219    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 07:00:00.179645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.179561    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"]
Nov 01 07:00:00.195902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.195860    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"]
Nov 01 07:00:00.196089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.195982    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.200772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.200750    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"]
Nov 01 07:00:00.200930 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.200884    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.204974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.204945    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.229242 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.229218    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"]
Nov 01 07:00:00.234147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.234018    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"]
Nov 01 07:00:00.234404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.234384    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.238807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.238783    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"]
Nov 01 07:00:00.238965 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.238911    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.243143 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.243122    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"]
Nov 01 07:00:00.243264 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.243231    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.247135 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.247112    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"]
Nov 01 07:00:00.247251 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.247238    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.251020 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.251002    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"]
Nov 01 07:00:00.251141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.251117    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.254948 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.254923    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"]
Nov 01 07:00:00.255041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.255024    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.258657 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.258637    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.261822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.261797    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmr6\" (UniqueName: \"kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6\") pod \"maas-api-key-cleanup-29891940-rtsnc\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.261970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.261953    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.262016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.261996    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fkg\" (UniqueName: \"kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.262111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.262089    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.262172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.262155    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pkk\" (UniqueName: \"kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.262220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.262199    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891940-rtsnc\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.275538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.275443    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:00:00.276527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.275442    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:00:00.276527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.275453    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:00:00.279741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.279718    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"]
Nov 01 07:00:00.284039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.284019    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.284354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.284326    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"]
Nov 01 07:00:00.289606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.289587    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"]
Nov 01 07:00:00.294361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.294338    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"]
Nov 01 07:00:00.333547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.333515    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"]
Nov 01 07:00:00.340509 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.340481    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"]
Nov 01 07:00:00.345585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.345556    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"]
Nov 01 07:00:00.363404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363367    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.363404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363408    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjqt\" (UniqueName: \"kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.363673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363517    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pkk\" (UniqueName: \"kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.363673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363547    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rx5m\" (UniqueName: \"kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m\") pod \"maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.363673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363570    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.363673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363618    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891940-rtsnc\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.363883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363794    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnl2\" (UniqueName: \"kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.363944 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363926    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.364026 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.363993    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmr6\" (UniqueName: \"kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6\") pod \"maas-api-key-cleanup-29891940-rtsnc\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.364111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364095    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnph8\" (UniqueName: \"kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8\") pod \"maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.364172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364120    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9tc\" (UniqueName: \"kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc\") pod \"maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.364172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364147    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.364280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364198    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4mt\" (UniqueName: \"kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.364280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364218    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgz9\" (UniqueName: \"kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.364384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364318    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891940-rtsnc\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.364384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364357    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.364524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364380    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.364524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364420    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.364524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364507    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.364701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364550    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fkg\" (UniqueName: \"kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.364701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364583    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvjb\" (UniqueName: \"kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.364701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364629    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.364701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.364660    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.365238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.365210    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.365358 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.365343    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.390176 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.390130    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"]
Nov 01 07:00:00.390331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.390219    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"]
Nov 01 07:00:00.398601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.398512    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"]
Nov 01 07:00:00.402105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.402077    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"]
Nov 01 07:00:00.420193 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.420159    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fkg\" (UniqueName: \"kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.420385 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.420371    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pkk\" (UniqueName: \"kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.420665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.420568    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmr6\" (UniqueName: \"kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6\") pod \"maas-api-key-cleanup-29891940-rtsnc\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.430257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.430181    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"]
Nov 01 07:00:00.466024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.465987    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.466024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466031    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.466265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466051    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjqt\" (UniqueName: \"kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.466265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466090    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rx5m\" (UniqueName: \"kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m\") pod \"maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.466265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466119    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.466265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466161    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnl2\" (UniqueName: \"kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.466265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466239    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.466568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466298    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnph8\" (UniqueName: \"kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8\") pod \"maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.466568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466327    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9tc\" (UniqueName: \"kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc\") pod \"maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.466568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466365    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.466568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466437    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4mt\" (UniqueName: \"kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.466568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466486    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgz9\" (UniqueName: \"kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.466568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466565    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.466891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466592    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.466891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466628    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.467001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466900    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvjb\" (UniqueName: \"kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.467001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.466952    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.467130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467106    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.467192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467112    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.467192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467142    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.467380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467360    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.467567 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467546    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.467676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467657    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.467734 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.467718    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.512226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.512191    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:00.520068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.520047    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:00.524222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.524194    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvjb\" (UniqueName: \"kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.524302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.524255    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9tc\" (UniqueName: \"kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc\") pod \"maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.524302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.524268    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjqt\" (UniqueName: \"kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.524302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.524262    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnph8\" (UniqueName: \"kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8\") pod \"maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.524866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.524844    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4mt\" (UniqueName: \"kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt\") pod \"maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.525624 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.525601    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnl2\" (UniqueName: \"kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.526031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.526007    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rx5m\" (UniqueName: \"kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m\") pod \"maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.526673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.526648    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:00.527115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.527094    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgz9\" (UniqueName: \"kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.552931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.552895    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:00.560487 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.560241    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:00.570887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.568517    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:00.576827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.576799    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:00.588532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.586390    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:00.594346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.594317    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:00.602910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.602722    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:00.611629 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.610990    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:00.931655 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.930970    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"]
Nov 01 07:00:00.932390 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:00.932357    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddbd24a_5ab5_42b7_bbfd_dc71a0f8bf82.slice/crio-f7a99dbc488b778564c66f98c8ca796a7a47c06f4d8f44b513974f2de0ec0fed WatchSource:0}: Error finding container f7a99dbc488b778564c66f98c8ca796a7a47c06f4d8f44b513974f2de0ec0fed: Status 404 returned error can't find the container with id f7a99dbc488b778564c66f98c8ca796a7a47c06f4d8f44b513974f2de0ec0fed
Nov 01 07:00:00.934016 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:00.933978    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2b8ad1_fc50_4904_89c5_5e801e0fd0a3.slice/crio-b459168707c826dfba61c4866a971890f819c8ecbec0e76facf9b4e14a944e45 WatchSource:0}: Error finding container b459168707c826dfba61c4866a971890f819c8ecbec0e76facf9b4e14a944e45: Status 404 returned error can't find the container with id b459168707c826dfba61c4866a971890f819c8ecbec0e76facf9b4e14a944e45
Nov 01 07:00:00.936139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.935036    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"]
Nov 01 07:00:00.978590 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.978560    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"]
Nov 01 07:00:00.987613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.987576    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerStarted","Data":"f7a99dbc488b778564c66f98c8ca796a7a47c06f4d8f44b513974f2de0ec0fed"}
Nov 01 07:00:00.988891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:00.988864    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerStarted","Data":"b459168707c826dfba61c4866a971890f819c8ecbec0e76facf9b4e14a944e45"}
Nov 01 07:00:01.008630 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.006909    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e690403_5f4d_479e_8a60_403ed4cb1cf6.slice/crio-8f8c3c39b41bf360c8e9e6a3c70fac2bd28b1b2d900a95977b2e9586bea6c21c WatchSource:0}: Error finding container 8f8c3c39b41bf360c8e9e6a3c70fac2bd28b1b2d900a95977b2e9586bea6c21c: Status 404 returned error can't find the container with id 8f8c3c39b41bf360c8e9e6a3c70fac2bd28b1b2d900a95977b2e9586bea6c21c
Nov 01 07:00:01.133327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.133290    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"]
Nov 01 07:00:01.139276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.139247    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"]
Nov 01 07:00:01.155864 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.155839    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"]
Nov 01 07:00:01.162432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.162404    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"]
Nov 01 07:00:01.207173 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.207144    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"]
Nov 01 07:00:01.207566 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.207540    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe39e287_82ea_4425_8b81_4b30381675d5.slice/crio-46402429170d3fadeccfa1f92bc64daca11300c02913bbdd709ff0146a428450 WatchSource:0}: Error finding container 46402429170d3fadeccfa1f92bc64daca11300c02913bbdd709ff0146a428450: Status 404 returned error can't find the container with id 46402429170d3fadeccfa1f92bc64daca11300c02913bbdd709ff0146a428450
Nov 01 07:00:01.208347 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.208326    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd8eaf6_11f3_40ad_aa1c_2e7ecb3f05c6.slice/crio-f1140f6395c877d40cfc932e0a4c97b55be422e4771707c4759bad778920be2f WatchSource:0}: Error finding container f1140f6395c877d40cfc932e0a4c97b55be422e4771707c4759bad778920be2f: Status 404 returned error can't find the container with id f1140f6395c877d40cfc932e0a4c97b55be422e4771707c4759bad778920be2f
Nov 01 07:00:01.209492 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.209293    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe256c8_4251_4a0f_9c7e_786bc5954f21.slice/crio-d8f108be109e89334bf01bdc99b95e2e7ff3b2510c1e70184657b062845bba3d WatchSource:0}: Error finding container d8f108be109e89334bf01bdc99b95e2e7ff3b2510c1e70184657b062845bba3d: Status 404 returned error can't find the container with id d8f108be109e89334bf01bdc99b95e2e7ff3b2510c1e70184657b062845bba3d
Nov 01 07:00:01.210662 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.210639    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717f3bb7_a336_4920_ac05_9c722a5bf9af.slice/crio-34e30c53a64a3dfb1ec5486235366ec5afe17c6d720cafbb3130ef8113eacab7 WatchSource:0}: Error finding container 34e30c53a64a3dfb1ec5486235366ec5afe17c6d720cafbb3130ef8113eacab7: Status 404 returned error can't find the container with id 34e30c53a64a3dfb1ec5486235366ec5afe17c6d720cafbb3130ef8113eacab7
Nov 01 07:00:01.212026 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.212006    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32836548_dfd4_4309_8897_095ab10d6909.slice/crio-52672370b8f8e3af5aa153b4246464beddda69d45a72808d3177d6301d6f8e17 WatchSource:0}: Error finding container 52672370b8f8e3af5aa153b4246464beddda69d45a72808d3177d6301d6f8e17: Status 404 returned error can't find the container with id 52672370b8f8e3af5aa153b4246464beddda69d45a72808d3177d6301d6f8e17
Nov 01 07:00:01.218929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.218881    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"]
Nov 01 07:00:01.223247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.223225    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"]
Nov 01 07:00:01.229411 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.226296    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"]
Nov 01 07:00:01.233215 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.233187    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0de880d_b6f4_45e4_b66e_1acd9c7ec1d7.slice/crio-cb6afe30be8dbf44fec0890c2c5d001cc7d688c101ef553e9cd342cc3b8aa76c WatchSource:0}: Error finding container cb6afe30be8dbf44fec0890c2c5d001cc7d688c101ef553e9cd342cc3b8aa76c: Status 404 returned error can't find the container with id cb6afe30be8dbf44fec0890c2c5d001cc7d688c101ef553e9cd342cc3b8aa76c
Nov 01 07:00:01.234257 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.234232    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1015013_8668_4ff1_8268_065840ae0a60.slice/crio-d854f8f21070b7bdcf3bd10323c171cd1fd2a87197f577c2ba84ddc86a777bb1 WatchSource:0}: Error finding container d854f8f21070b7bdcf3bd10323c171cd1fd2a87197f577c2ba84ddc86a777bb1: Status 404 returned error can't find the container with id d854f8f21070b7bdcf3bd10323c171cd1fd2a87197f577c2ba84ddc86a777bb1
Nov 01 07:00:01.236291 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:01.236269    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8787183_583d_4e63_be9d_8b211b87d097.slice/crio-7a2d88e8e155f6cb5b2ea3ec26d1b1e718856b4f2b0fb07cc5b2313667c2e2ce WatchSource:0}: Error finding container 7a2d88e8e155f6cb5b2ea3ec26d1b1e718856b4f2b0fb07cc5b2313667c2e2ce: Status 404 returned error can't find the container with id 7a2d88e8e155f6cb5b2ea3ec26d1b1e718856b4f2b0fb07cc5b2313667c2e2ce
Nov 01 07:00:01.996945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.996909    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerStarted","Data":"e992a8ebed86ea48176c26222f8d7d8a8b81f568076a9c2c8b61f6b26ed08c1d"}
Nov 01 07:00:01.997144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.996965    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerStarted","Data":"d854f8f21070b7bdcf3bd10323c171cd1fd2a87197f577c2ba84ddc86a777bb1"}
Nov 01 07:00:01.998577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.998546    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerStarted","Data":"fbe51024553202202ef36aee732ce11351c6f51f4aead5d86373401e04299354"}
Nov 01 07:00:01.998705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:01.998598    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerStarted","Data":"46402429170d3fadeccfa1f92bc64daca11300c02913bbdd709ff0146a428450"}
Nov 01 07:00:02.000105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.000083    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerStarted","Data":"9c7f0c4d21fdfcdffe48ab4ff8d9b682b79815d1e67f52ddc97520b248f088b8"}
Nov 01 07:00:02.000257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.000133    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerStarted","Data":"52672370b8f8e3af5aa153b4246464beddda69d45a72808d3177d6301d6f8e17"}
Nov 01 07:00:02.001589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.001568    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerStarted","Data":"ec84c8d3eb7b86f5b323d0af63b8758dc2bf306279d7b7ac7674f2206eec0b01"}
Nov 01 07:00:02.001670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.001610    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerStarted","Data":"d8f108be109e89334bf01bdc99b95e2e7ff3b2510c1e70184657b062845bba3d"}
Nov 01 07:00:02.007603 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.007578    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerStarted","Data":"e1129335b8303aafc4b8f6ed21b9bd2441bd775e8a0c6cbb8d703f75e3646c9f"}
Nov 01 07:00:02.007708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.007617    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerStarted","Data":"cb6afe30be8dbf44fec0890c2c5d001cc7d688c101ef553e9cd342cc3b8aa76c"}
Nov 01 07:00:02.009364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.009340    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerStarted","Data":"ed807fc98372ab7816e6fdaaec0270c3e7a9351e42935102b7ec1857a7961034"}
Nov 01 07:00:02.009497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.009378    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerStarted","Data":"7a2d88e8e155f6cb5b2ea3ec26d1b1e718856b4f2b0fb07cc5b2313667c2e2ce"}
Nov 01 07:00:02.011150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.011126    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerStarted","Data":"5792f91480735b78f43b4d9bad1558f4512d87cdad0bccdee3b426e3039f6e79"}
Nov 01 07:00:02.012685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.012664    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerStarted","Data":"786a8d4372ea03414ce4cff5b6f66292ddea8ae413619ba325bd34c16d99dbdd"}
Nov 01 07:00:02.012793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.012701    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerStarted","Data":"f1140f6395c877d40cfc932e0a4c97b55be422e4771707c4759bad778920be2f"}
Nov 01 07:00:02.014079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.014056    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerStarted","Data":"bb6362077977a30929660ab4c544e4566a03d038b73f5988ce71894cee0760f5"}
Nov 01 07:00:02.015451 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.015431    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerStarted","Data":"8cf41a15bb82f35a98305d200b207adbcf259afbd5d01bb70df26a0b55bfa97c"}
Nov 01 07:00:02.015566 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.015512    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerStarted","Data":"8f8c3c39b41bf360c8e9e6a3c70fac2bd28b1b2d900a95977b2e9586bea6c21c"}
Nov 01 07:00:02.016824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.016801    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerStarted","Data":"af203a93ff7d27bd48ff5b635cb1884d94df1cd814f2c9d8b13e94788d0fc2ef"}
Nov 01 07:00:02.016891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.016841    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerStarted","Data":"34e30c53a64a3dfb1ec5486235366ec5afe17c6d720cafbb3130ef8113eacab7"}
Nov 01 07:00:02.110363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.110107    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" podStartSLOduration=2.110089163 podStartE2EDuration="2.110089163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.1086751 +0000 UTC m=+3013.676481931" watchObservedRunningTime="2026-11-01 07:00:02.110089163 +0000 UTC m=+3013.677895995"
Nov 01 07:00:02.223678 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.223629    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" podStartSLOduration=2.223615052 podStartE2EDuration="2.223615052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.219547176 +0000 UTC m=+3013.787354006" watchObservedRunningTime="2026-11-01 07:00:02.223615052 +0000 UTC m=+3013.791421881"
Nov 01 07:00:02.336919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.336870    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" podStartSLOduration=2.336856554 podStartE2EDuration="2.336856554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.334160843 +0000 UTC m=+3013.901967687" watchObservedRunningTime="2026-11-01 07:00:02.336856554 +0000 UTC m=+3013.904663382"
Nov 01 07:00:02.446857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.446799    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" podStartSLOduration=2.446783531 podStartE2EDuration="2.446783531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.444809597 +0000 UTC m=+3014.012616440" watchObservedRunningTime="2026-11-01 07:00:02.446783531 +0000 UTC m=+3014.014590416"
Nov 01 07:00:02.561805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.561755    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" podStartSLOduration=2.561741118 podStartE2EDuration="2.561741118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.558444922 +0000 UTC m=+3014.126251773" watchObservedRunningTime="2026-11-01 07:00:02.561741118 +0000 UTC m=+3014.129547943"
Nov 01 07:00:02.671456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.671394    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" podStartSLOduration=2.671381249 podStartE2EDuration="2.671381249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.670059433 +0000 UTC m=+3014.237866262" watchObservedRunningTime="2026-11-01 07:00:02.671381249 +0000 UTC m=+3014.239188115"
Nov 01 07:00:02.783440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.783389    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" podStartSLOduration=2.783373816 podStartE2EDuration="2.783373816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.780343953 +0000 UTC m=+3014.348150785" watchObservedRunningTime="2026-11-01 07:00:02.783373816 +0000 UTC m=+3014.351180673"
Nov 01 07:00:02.894008 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:02.893939    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" podStartSLOduration=2.893918586 podStartE2EDuration="2.893918586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.890679617 +0000 UTC m=+3014.458486446" watchObservedRunningTime="2026-11-01 07:00:02.893918586 +0000 UTC m=+3014.461725415"
Nov 01 07:00:02.948962 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:02.948929    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:00:03.004585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.004483    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" podStartSLOduration=3.004451821 podStartE2EDuration="3.004451821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:03.001925534 +0000 UTC m=+3014.569732368" watchObservedRunningTime="2026-11-01 07:00:03.004451821 +0000 UTC m=+3014.572258687"
Nov 01 07:00:03.025768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.025725    4468 generic.go:356] "Generic (PLEG): container finished" podID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerID="bb6362077977a30929660ab4c544e4566a03d038b73f5988ce71894cee0760f5" exitCode=60
Nov 01 07:00:03.025947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.025840    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerDied","Data":"bb6362077977a30929660ab4c544e4566a03d038b73f5988ce71894cee0760f5"}
Nov 01 07:00:03.026238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.026213    4468 scope.go:122] "RemoveContainer" containerID="bb6362077977a30929660ab4c544e4566a03d038b73f5988ce71894cee0760f5"
Nov 01 07:00:03.027878 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.027854    4468 generic.go:356] "Generic (PLEG): container finished" podID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerID="8cf41a15bb82f35a98305d200b207adbcf259afbd5d01bb70df26a0b55bfa97c" exitCode=60
Nov 01 07:00:03.028036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.027961    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerDied","Data":"8cf41a15bb82f35a98305d200b207adbcf259afbd5d01bb70df26a0b55bfa97c"}
Nov 01 07:00:03.028195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.028179    4468 scope.go:122] "RemoveContainer" containerID="8cf41a15bb82f35a98305d200b207adbcf259afbd5d01bb70df26a0b55bfa97c"
Nov 01 07:00:03.029714 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.029674    4468 generic.go:356] "Generic (PLEG): container finished" podID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerID="af203a93ff7d27bd48ff5b635cb1884d94df1cd814f2c9d8b13e94788d0fc2ef" exitCode=60
Nov 01 07:00:03.029785 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.029695    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerDied","Data":"af203a93ff7d27bd48ff5b635cb1884d94df1cd814f2c9d8b13e94788d0fc2ef"}
Nov 01 07:00:03.029951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.029939    4468 scope.go:122] "RemoveContainer" containerID="af203a93ff7d27bd48ff5b635cb1884d94df1cd814f2c9d8b13e94788d0fc2ef"
Nov 01 07:00:03.031612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.031593    4468 generic.go:356] "Generic (PLEG): container finished" podID="a1015013-8668-4ff1-8268-065840ae0a60" containerID="e992a8ebed86ea48176c26222f8d7d8a8b81f568076a9c2c8b61f6b26ed08c1d" exitCode=60
Nov 01 07:00:03.031681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.031659    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerDied","Data":"e992a8ebed86ea48176c26222f8d7d8a8b81f568076a9c2c8b61f6b26ed08c1d"}
Nov 01 07:00:03.031832 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.031818    4468 scope.go:122] "RemoveContainer" containerID="e992a8ebed86ea48176c26222f8d7d8a8b81f568076a9c2c8b61f6b26ed08c1d"
Nov 01 07:00:03.033302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.033275    4468 generic.go:356] "Generic (PLEG): container finished" podID="be39e287-82ea-4425-8b81-4b30381675d5" containerID="fbe51024553202202ef36aee732ce11351c6f51f4aead5d86373401e04299354" exitCode=60
Nov 01 07:00:03.033368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.033344    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerDied","Data":"fbe51024553202202ef36aee732ce11351c6f51f4aead5d86373401e04299354"}
Nov 01 07:00:03.033547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.033534    4468 scope.go:122] "RemoveContainer" containerID="fbe51024553202202ef36aee732ce11351c6f51f4aead5d86373401e04299354"
Nov 01 07:00:03.034931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.034913    4468 generic.go:356] "Generic (PLEG): container finished" podID="32836548-dfd4-4309-8897-095ab10d6909" containerID="9c7f0c4d21fdfcdffe48ab4ff8d9b682b79815d1e67f52ddc97520b248f088b8" exitCode=60
Nov 01 07:00:03.035015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.034981    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerDied","Data":"9c7f0c4d21fdfcdffe48ab4ff8d9b682b79815d1e67f52ddc97520b248f088b8"}
Nov 01 07:00:03.035155 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.035143    4468 scope.go:122] "RemoveContainer" containerID="9c7f0c4d21fdfcdffe48ab4ff8d9b682b79815d1e67f52ddc97520b248f088b8"
Nov 01 07:00:03.036658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.036637    4468 generic.go:356] "Generic (PLEG): container finished" podID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerID="ec84c8d3eb7b86f5b323d0af63b8758dc2bf306279d7b7ac7674f2206eec0b01" exitCode=60
Nov 01 07:00:03.036733 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.036700    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerDied","Data":"ec84c8d3eb7b86f5b323d0af63b8758dc2bf306279d7b7ac7674f2206eec0b01"}
Nov 01 07:00:03.036940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.036920    4468 scope.go:122] "RemoveContainer" containerID="ec84c8d3eb7b86f5b323d0af63b8758dc2bf306279d7b7ac7674f2206eec0b01"
Nov 01 07:00:03.038425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.038407    4468 generic.go:356] "Generic (PLEG): container finished" podID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerID="e1129335b8303aafc4b8f6ed21b9bd2441bd775e8a0c6cbb8d703f75e3646c9f" exitCode=60
Nov 01 07:00:03.038575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.038555    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerDied","Data":"e1129335b8303aafc4b8f6ed21b9bd2441bd775e8a0c6cbb8d703f75e3646c9f"}
Nov 01 07:00:03.038744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.038723    4468 scope.go:122] "RemoveContainer" containerID="e1129335b8303aafc4b8f6ed21b9bd2441bd775e8a0c6cbb8d703f75e3646c9f"
Nov 01 07:00:03.040318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.040296    4468 generic.go:356] "Generic (PLEG): container finished" podID="a8787183-583d-4e63-be9d-8b211b87d097" containerID="ed807fc98372ab7816e6fdaaec0270c3e7a9351e42935102b7ec1857a7961034" exitCode=60
Nov 01 07:00:03.040389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.040369    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerDied","Data":"ed807fc98372ab7816e6fdaaec0270c3e7a9351e42935102b7ec1857a7961034"}
Nov 01 07:00:03.040639 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.040624    4468 scope.go:122] "RemoveContainer" containerID="ed807fc98372ab7816e6fdaaec0270c3e7a9351e42935102b7ec1857a7961034"
Nov 01 07:00:03.042030 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.042010    4468 generic.go:356] "Generic (PLEG): container finished" podID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerID="5792f91480735b78f43b4d9bad1558f4512d87cdad0bccdee3b426e3039f6e79" exitCode=60
Nov 01 07:00:03.042354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.042238    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerDied","Data":"5792f91480735b78f43b4d9bad1558f4512d87cdad0bccdee3b426e3039f6e79"}
Nov 01 07:00:03.042432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.042356    4468 scope.go:122] "RemoveContainer" containerID="5792f91480735b78f43b4d9bad1558f4512d87cdad0bccdee3b426e3039f6e79"
Nov 01 07:00:03.043806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.043785    4468 generic.go:356] "Generic (PLEG): container finished" podID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerID="786a8d4372ea03414ce4cff5b6f66292ddea8ae413619ba325bd34c16d99dbdd" exitCode=60
Nov 01 07:00:03.043884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.043832    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerDied","Data":"786a8d4372ea03414ce4cff5b6f66292ddea8ae413619ba325bd34c16d99dbdd"}
Nov 01 07:00:03.044030 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.044017    4468 scope.go:122] "RemoveContainer" containerID="786a8d4372ea03414ce4cff5b6f66292ddea8ae413619ba325bd34c16d99dbdd"
Nov 01 07:00:03.115425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.115381    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" podStartSLOduration=3.115366663 podStartE2EDuration="3.115366663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:03.113619004 +0000 UTC m=+3014.681425856" watchObservedRunningTime="2026-11-01 07:00:03.115366663 +0000 UTC m=+3014.683173526"
Nov 01 07:00:03.228012 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:03.227949    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" podStartSLOduration=3.227932841 podStartE2EDuration="3.227932841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:03.227110262 +0000 UTC m=+3014.794917092" watchObservedRunningTime="2026-11-01 07:00:03.227932841 +0000 UTC m=+3014.795739671"
Nov 01 07:00:04.052497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.052430    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerStarted","Data":"38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40"}
Nov 01 07:00:04.054608 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.054576    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerStarted","Data":"2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b"}
Nov 01 07:00:04.056479 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.056444    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerStarted","Data":"2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd"}
Nov 01 07:00:04.058289 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.058261    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerStarted","Data":"5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7"}
Nov 01 07:00:04.060114 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.060084    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerStarted","Data":"419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c"}
Nov 01 07:00:04.062085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.062066    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerStarted","Data":"fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522"}
Nov 01 07:00:04.063886 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.063863    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerStarted","Data":"69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2"}
Nov 01 07:00:04.065692 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.065669    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerStarted","Data":"20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a"}
Nov 01 07:00:04.067327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.067307    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerStarted","Data":"9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0"}
Nov 01 07:00:04.069137 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.069117    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerStarted","Data":"c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef"}
Nov 01 07:00:04.070829 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:04.070812    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerStarted","Data":"0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd"}
Nov 01 07:00:05.080315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.080277    4468 generic.go:356] "Generic (PLEG): container finished" podID="32836548-dfd4-4309-8897-095ab10d6909" containerID="2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd" exitCode=60
Nov 01 07:00:05.080811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.080364    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerDied","Data":"2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd"}
Nov 01 07:00:05.080811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.080447    4468 scope.go:122] "RemoveContainer" containerID="9c7f0c4d21fdfcdffe48ab4ff8d9b682b79815d1e67f52ddc97520b248f088b8"
Nov 01 07:00:05.080811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.080765    4468 scope.go:122] "RemoveContainer" containerID="2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd"
Nov 01 07:00:05.081197 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.081160    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56_odh-ai-gateway-infra(32836548-dfd4-4309-8897-095ab10d6909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" podUID="32836548-dfd4-4309-8897-095ab10d6909"
Nov 01 07:00:05.082726 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.082704    4468 generic.go:356] "Generic (PLEG): container finished" podID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerID="5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7" exitCode=60
Nov 01 07:00:05.082816 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.082775    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerDied","Data":"5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7"}
Nov 01 07:00:05.083144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.083125    4468 scope.go:122] "RemoveContainer" containerID="5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7"
Nov 01 07:00:05.083632 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.083608    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42_odh-ai-gateway-infra(dfe256c8-4251-4a0f-9c7e-786bc5954f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21"
Nov 01 07:00:05.085424 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.085297    4468 generic.go:356] "Generic (PLEG): container finished" podID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerID="419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c" exitCode=60
Nov 01 07:00:05.085424 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.085388    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerDied","Data":"419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c"}
Nov 01 07:00:05.085808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.085767    4468 scope.go:122] "RemoveContainer" containerID="419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c"
Nov 01 07:00:05.086287 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.086139    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d_odh-ai-gateway-infra(b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7"
Nov 01 07:00:05.087669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.087644    4468 generic.go:356] "Generic (PLEG): container finished" podID="a8787183-583d-4e63-be9d-8b211b87d097" containerID="fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522" exitCode=60
Nov 01 07:00:05.087779 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.087732    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerDied","Data":"fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522"}
Nov 01 07:00:05.088000 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.087980    4468 scope.go:122] "RemoveContainer" containerID="fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522"
Nov 01 07:00:05.088299 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.088274    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx_odh-ai-gateway-infra(a8787183-583d-4e63-be9d-8b211b87d097)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" podUID="a8787183-583d-4e63-be9d-8b211b87d097"
Nov 01 07:00:05.089938 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.089905    4468 generic.go:356] "Generic (PLEG): container finished" podID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerID="69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2" exitCode=60
Nov 01 07:00:05.090043 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.089949    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerDied","Data":"69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2"}
Nov 01 07:00:05.090251 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.090233    4468 scope.go:122] "RemoveContainer" containerID="69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2"
Nov 01 07:00:05.090597 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.090573    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms_odh-ai-gateway-infra(7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82"
Nov 01 07:00:05.092096 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.092076    4468 generic.go:356] "Generic (PLEG): container finished" podID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerID="20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a" exitCode=60
Nov 01 07:00:05.092178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.092156    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerDied","Data":"20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a"}
Nov 01 07:00:05.092371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.092357    4468 scope.go:122] "RemoveContainer" containerID="20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a"
Nov 01 07:00:05.092643 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.092621    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl_odh-ai-gateway-infra(1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6"
Nov 01 07:00:05.094524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.094500    4468 generic.go:356] "Generic (PLEG): container finished" podID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerID="9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0" exitCode=60
Nov 01 07:00:05.094633 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.094585    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerDied","Data":"9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0"}
Nov 01 07:00:05.094799 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.094785    4468 scope.go:122] "RemoveContainer" containerID="9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0"
Nov 01 07:00:05.095062 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.095038    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891940-rtsnc_odh-ai-gateway-infra(5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3"
Nov 01 07:00:05.095796 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.095780    4468 scope.go:122] "RemoveContainer" containerID="ec84c8d3eb7b86f5b323d0af63b8758dc2bf306279d7b7ac7674f2206eec0b01"
Nov 01 07:00:05.096814 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.096791    4468 generic.go:356] "Generic (PLEG): container finished" podID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerID="c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef" exitCode=60
Nov 01 07:00:05.096935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.096903    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerDied","Data":"c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef"}
Nov 01 07:00:05.097068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.097060    4468 scope.go:122] "RemoveContainer" containerID="c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef"
Nov 01 07:00:05.097336 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.097313    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc_odh-ai-gateway-infra(4e690403-5f4d-479e-8a60-403ed4cb1cf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6"
Nov 01 07:00:05.099276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.099254    4468 generic.go:356] "Generic (PLEG): container finished" podID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerID="0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd" exitCode=60
Nov 01 07:00:05.099380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.099329    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerDied","Data":"0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd"}
Nov 01 07:00:05.099544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.099528    4468 scope.go:122] "RemoveContainer" containerID="0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd"
Nov 01 07:00:05.099801 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.099780    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv_odh-ai-gateway-infra(717f3bb7-a336-4920-ac05-9c722a5bf9af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af"
Nov 01 07:00:05.101238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.101215    4468 generic.go:356] "Generic (PLEG): container finished" podID="a1015013-8668-4ff1-8268-065840ae0a60" containerID="38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40" exitCode=60
Nov 01 07:00:05.101345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.101292    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerDied","Data":"38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40"}
Nov 01 07:00:05.101526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.101511    4468 scope.go:122] "RemoveContainer" containerID="38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40"
Nov 01 07:00:05.101732 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.101708    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw_odh-ai-gateway-infra(a1015013-8668-4ff1-8268-065840ae0a60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" podUID="a1015013-8668-4ff1-8268-065840ae0a60"
Nov 01 07:00:05.103522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.103501    4468 generic.go:356] "Generic (PLEG): container finished" podID="be39e287-82ea-4425-8b81-4b30381675d5" containerID="2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b" exitCode=60
Nov 01 07:00:05.103606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.103574    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerDied","Data":"2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b"}
Nov 01 07:00:05.103757 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.103744    4468 scope.go:122] "RemoveContainer" containerID="2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b"
Nov 01 07:00:05.103948 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:05.103933    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch_odh-ai-gateway-infra(be39e287-82ea-4425-8b81-4b30381675d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" podUID="be39e287-82ea-4425-8b81-4b30381675d5"
Nov 01 07:00:05.110307 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.110286    4468 scope.go:122] "RemoveContainer" containerID="e1129335b8303aafc4b8f6ed21b9bd2441bd775e8a0c6cbb8d703f75e3646c9f"
Nov 01 07:00:05.126134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.126110    4468 scope.go:122] "RemoveContainer" containerID="ed807fc98372ab7816e6fdaaec0270c3e7a9351e42935102b7ec1857a7961034"
Nov 01 07:00:05.141846 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.141818    4468 scope.go:122] "RemoveContainer" containerID="5792f91480735b78f43b4d9bad1558f4512d87cdad0bccdee3b426e3039f6e79"
Nov 01 07:00:05.158345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.158316    4468 scope.go:122] "RemoveContainer" containerID="786a8d4372ea03414ce4cff5b6f66292ddea8ae413619ba325bd34c16d99dbdd"
Nov 01 07:00:05.172387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.172364    4468 scope.go:122] "RemoveContainer" containerID="bb6362077977a30929660ab4c544e4566a03d038b73f5988ce71894cee0760f5"
Nov 01 07:00:05.187297 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.187270    4468 scope.go:122] "RemoveContainer" containerID="8cf41a15bb82f35a98305d200b207adbcf259afbd5d01bb70df26a0b55bfa97c"
Nov 01 07:00:05.202621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.202597    4468 scope.go:122] "RemoveContainer" containerID="af203a93ff7d27bd48ff5b635cb1884d94df1cd814f2c9d8b13e94788d0fc2ef"
Nov 01 07:00:05.215621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.215599    4468 scope.go:122] "RemoveContainer" containerID="e992a8ebed86ea48176c26222f8d7d8a8b81f568076a9c2c8b61f6b26ed08c1d"
Nov 01 07:00:05.228873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:05.228850    4468 scope.go:122] "RemoveContainer" containerID="fbe51024553202202ef36aee732ce11351c6f51f4aead5d86373401e04299354"
Nov 01 07:00:06.113512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.113453    4468 scope.go:122] "RemoveContainer" containerID="9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0"
Nov 01 07:00:06.114103 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.113862    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891940-rtsnc_odh-ai-gateway-infra(5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3"
Nov 01 07:00:06.115238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.115215    4468 scope.go:122] "RemoveContainer" containerID="c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef"
Nov 01 07:00:06.115596 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.115574    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc_odh-ai-gateway-infra(4e690403-5f4d-479e-8a60-403ed4cb1cf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6"
Nov 01 07:00:06.116956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.116937    4468 scope.go:122] "RemoveContainer" containerID="0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd"
Nov 01 07:00:06.117235 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.117217    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv_odh-ai-gateway-infra(717f3bb7-a336-4920-ac05-9c722a5bf9af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af"
Nov 01 07:00:06.118582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.118562    4468 scope.go:122] "RemoveContainer" containerID="38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40"
Nov 01 07:00:06.118872 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.118852    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw_odh-ai-gateway-infra(a1015013-8668-4ff1-8268-065840ae0a60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" podUID="a1015013-8668-4ff1-8268-065840ae0a60"
Nov 01 07:00:06.120173 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.120157    4468 scope.go:122] "RemoveContainer" containerID="2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b"
Nov 01 07:00:06.120405 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.120384    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch_odh-ai-gateway-infra(be39e287-82ea-4425-8b81-4b30381675d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" podUID="be39e287-82ea-4425-8b81-4b30381675d5"
Nov 01 07:00:06.121863 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.121847    4468 scope.go:122] "RemoveContainer" containerID="2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd"
Nov 01 07:00:06.122078 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.122058    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56_odh-ai-gateway-infra(32836548-dfd4-4309-8897-095ab10d6909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" podUID="32836548-dfd4-4309-8897-095ab10d6909"
Nov 01 07:00:06.123366 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.123349    4468 scope.go:122] "RemoveContainer" containerID="5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7"
Nov 01 07:00:06.123618 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.123594    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42_odh-ai-gateway-infra(dfe256c8-4251-4a0f-9c7e-786bc5954f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21"
Nov 01 07:00:06.124808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.124791    4468 scope.go:122] "RemoveContainer" containerID="419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c"
Nov 01 07:00:06.125020 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.124995    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d_odh-ai-gateway-infra(b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7"
Nov 01 07:00:06.126388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.126368    4468 scope.go:122] "RemoveContainer" containerID="fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522"
Nov 01 07:00:06.126626 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.126603    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx_odh-ai-gateway-infra(a8787183-583d-4e63-be9d-8b211b87d097)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" podUID="a8787183-583d-4e63-be9d-8b211b87d097"
Nov 01 07:00:06.128171 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.128154    4468 scope.go:122] "RemoveContainer" containerID="69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2"
Nov 01 07:00:06.128354 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.128337    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms_odh-ai-gateway-infra(7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82"
Nov 01 07:00:06.129696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.129680    4468 scope.go:122] "RemoveContainer" containerID="20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a"
Nov 01 07:00:06.129871 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:06.129850    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl_odh-ai-gateway-infra(1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6"
Nov 01 07:00:06.149429 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.149174    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"]
Nov 01 07:00:06.155063 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.155045    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.173923 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.173888    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48x6v\" (UniqueName: \"kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.174067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.173935    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.174402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.174383    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.209009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.208977    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:00:06.248198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.245050    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"]
Nov 01 07:00:06.276258 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.276220    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48x6v\" (UniqueName: \"kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.276446 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.276270    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.276531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.276482    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.277068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.277042    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.278905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.278885    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.333159 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.333130    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48x6v\" (UniqueName: \"kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v\") pod \"maas-api-revoke-sub-4c8875ea-x88jt\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.468937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.468851    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:06.846145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:06.846107    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"]
Nov 01 07:00:06.847569 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:06.847533    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7452fc_4b86_416e_ac9e_2c1b29b2d1ae.slice/crio-2c6a4231810d50fba272a924a8233aec11a2dc5f43b4858c80e5d09b9a6de413 WatchSource:0}: Error finding container 2c6a4231810d50fba272a924a8233aec11a2dc5f43b4858c80e5d09b9a6de413: Status 404 returned error can't find the container with id 2c6a4231810d50fba272a924a8233aec11a2dc5f43b4858c80e5d09b9a6de413
Nov 01 07:00:07.138315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:07.138277    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt" event={"ID":"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae","Type":"ContainerStarted","Data":"92a605da908fcbef32a58418651349ef5a0228daf5071daa5eac0d73f445e32c"}
Nov 01 07:00:07.138770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:07.138366    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt" event={"ID":"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae","Type":"ContainerStarted","Data":"2c6a4231810d50fba272a924a8233aec11a2dc5f43b4858c80e5d09b9a6de413"}
Nov 01 07:00:08.147094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:08.147058    4468 generic.go:356] "Generic (PLEG): container finished" podID="5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" containerID="92a605da908fcbef32a58418651349ef5a0228daf5071daa5eac0d73f445e32c" exitCode=0
Nov 01 07:00:08.147611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:08.147144    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt" event={"ID":"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae","Type":"ContainerDied","Data":"92a605da908fcbef32a58418651349ef5a0228daf5071daa5eac0d73f445e32c"}
Nov 01 07:00:09.302962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.302931    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:09.434841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.434752    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v\" (UniqueName: \"kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v\") pod \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") "
Nov 01 07:00:09.435019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.434859    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca\") pod \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") "
Nov 01 07:00:09.435096 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.435030    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca\") pod \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\" (UID: \"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae\") "
Nov 01 07:00:09.435425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.435395    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca" pod "5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" (UID: "5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:09.436177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.436139    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:09.437215 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.437183    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v" pod "5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" (UID: "5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae"). InnerVolumeSpecName "kube-api-access-48x6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:09.437806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.437778    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca" pod "5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" (UID: "5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:00:09.537283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.537251    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:09.537283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:09.537281    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48x6v\" (UniqueName: \"kubernetes.io/projected/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae-kube-api-access-48x6v\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:10.162228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:10.162181    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt" event={"ID":"5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae","Type":"ContainerDied","Data":"2c6a4231810d50fba272a924a8233aec11a2dc5f43b4858c80e5d09b9a6de413"}
Nov 01 07:00:10.162228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:10.162222    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"
Nov 01 07:00:10.162415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:10.162241    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c6a4231810d50fba272a924a8233aec11a2dc5f43b4858c80e5d09b9a6de413"
Nov 01 07:00:15.583851 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:15.583820    4468 scope.go:122] "RemoveContainer" containerID="2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b"
Nov 01 07:00:16.207349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.207268    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerStarted","Data":"4e81ad1e4aaaa63413cf45c53fc6afdae69d183145b6d1bf384370f8d665f129"}
Nov 01 07:00:16.530022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.529932    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:00:16.582829 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.582796    4468 scope.go:122] "RemoveContainer" containerID="38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40"
Nov 01 07:00:16.583062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.583027    4468 scope.go:122] "RemoveContainer" containerID="69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2"
Nov 01 07:00:16.583196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.583177    4468 scope.go:122] "RemoveContainer" containerID="fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522"
Nov 01 07:00:16.949084 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.949056    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"]
Nov 01 07:00:16.949725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.949711    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" containerName="revoke-keys"
Nov 01 07:00:16.949780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.949726    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" containerName="revoke-keys"
Nov 01 07:00:16.949866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.949856    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" containerName="revoke-keys"
Nov 01 07:00:16.953437 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:16.953419    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.006210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.006184    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:00:17.013809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.012546    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"]
Nov 01 07:00:17.044331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.044304    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.044538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.044518    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67z8q\" (UniqueName: \"kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.044744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.044722    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.146194 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.146165    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.146369 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.146220    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.146369 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.146340    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z8q\" (UniqueName: \"kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.146834 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.146813    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.148663 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.148642    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.202908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.202846    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z8q\" (UniqueName: \"kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q\") pod \"maas-api-revoke-sub-a010bc65-g5cq2\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.217025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.216994    4468 generic.go:356] "Generic (PLEG): container finished" podID="be39e287-82ea-4425-8b81-4b30381675d5" containerID="4e81ad1e4aaaa63413cf45c53fc6afdae69d183145b6d1bf384370f8d665f129" exitCode=60
Nov 01 07:00:17.217141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.217066    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerDied","Data":"4e81ad1e4aaaa63413cf45c53fc6afdae69d183145b6d1bf384370f8d665f129"}
Nov 01 07:00:17.217141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.217138    4468 scope.go:122] "RemoveContainer" containerID="2212e6aaa88636fbe33956625e9fe9d4a5c5ba48c314fb964c34462925ebc51b"
Nov 01 07:00:17.217568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.217550    4468 scope.go:122] "RemoveContainer" containerID="4e81ad1e4aaaa63413cf45c53fc6afdae69d183145b6d1bf384370f8d665f129"
Nov 01 07:00:17.217968 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:17.217939    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch_odh-ai-gateway-infra(be39e287-82ea-4425-8b81-4b30381675d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" podUID="be39e287-82ea-4425-8b81-4b30381675d5"
Nov 01 07:00:17.219484 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.219443    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerStarted","Data":"83360a584c931667c0b5cc0b37c0264138ea12b4ecf92cdc432c991b78793937"}
Nov 01 07:00:17.221809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.221787    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerStarted","Data":"554841df0ac7b746928d0a01cc156ab227e60f644258b22396d80a6ee12e3752"}
Nov 01 07:00:17.224919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.224894    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerStarted","Data":"f29898ed5df7a386a788d23cf6b08adadced37979c1148e897a8bb6b221054af"}
Nov 01 07:00:17.265974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.265952    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:17.336091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.335257    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"]
Nov 01 07:00:17.583084 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.583055    4468 scope.go:122] "RemoveContainer" containerID="c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef"
Nov 01 07:00:17.583274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.583257    4468 scope.go:122] "RemoveContainer" containerID="0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd"
Nov 01 07:00:17.583592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.583573    4468 scope.go:122] "RemoveContainer" containerID="419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c"
Nov 01 07:00:17.583676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.583662    4468 scope.go:122] "RemoveContainer" containerID="5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7"
Nov 01 07:00:17.646202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:17.646178    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"]
Nov 01 07:00:17.713348 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:17.713321    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae53244a_80f1_4f4b_823a_dad36025b759.slice/crio-4d60836423885b8b7cf7a44553013ddde2274ab76ff9b81138bcc824481b2708 WatchSource:0}: Error finding container 4d60836423885b8b7cf7a44553013ddde2274ab76ff9b81138bcc824481b2708: Status 404 returned error can't find the container with id 4d60836423885b8b7cf7a44553013ddde2274ab76ff9b81138bcc824481b2708
Nov 01 07:00:18.233731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.233703    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerStarted","Data":"832dafa22bfa3a93037cd0ed56fc7f0f05e3efdb0cb935ce7799a5f240eac6a8"}
Nov 01 07:00:18.235583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.235556    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerStarted","Data":"4bfee6d05e3f94a808e6c84163f22909ee70c5dbcf0b89290915767b1b13fb6c"}
Nov 01 07:00:18.237014 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.236989    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2" event={"ID":"ae53244a-80f1-4f4b-823a-dad36025b759","Type":"ContainerStarted","Data":"ef588fd6cfa974bd0d8fb95c57e2e21a24dc6030f16998059ef6ab2b5e1d91e8"}
Nov 01 07:00:18.237144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.237033    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2" event={"ID":"ae53244a-80f1-4f4b-823a-dad36025b759","Type":"ContainerStarted","Data":"4d60836423885b8b7cf7a44553013ddde2274ab76ff9b81138bcc824481b2708"}
Nov 01 07:00:18.238892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.238873    4468 generic.go:356] "Generic (PLEG): container finished" podID="a1015013-8668-4ff1-8268-065840ae0a60" containerID="f29898ed5df7a386a788d23cf6b08adadced37979c1148e897a8bb6b221054af" exitCode=60
Nov 01 07:00:18.238987 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.238940    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerDied","Data":"f29898ed5df7a386a788d23cf6b08adadced37979c1148e897a8bb6b221054af"}
Nov 01 07:00:18.238987 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.238983    4468 scope.go:122] "RemoveContainer" containerID="38ba0807a7735f74baf04bc92c3e11f31b5ea6ada76982e53974736496d42d40"
Nov 01 07:00:18.239295 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.239276    4468 scope.go:122] "RemoveContainer" containerID="f29898ed5df7a386a788d23cf6b08adadced37979c1148e897a8bb6b221054af"
Nov 01 07:00:18.239590 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:18.239566    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw_odh-ai-gateway-infra(a1015013-8668-4ff1-8268-065840ae0a60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" podUID="a1015013-8668-4ff1-8268-065840ae0a60"
Nov 01 07:00:18.242371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.242352    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerStarted","Data":"69cb69cd1dfba0e16ce2b6398a053fae70cba766b727b53936738af443e7746b"}
Nov 01 07:00:18.244290 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.244268    4468 generic.go:356] "Generic (PLEG): container finished" podID="a8787183-583d-4e63-be9d-8b211b87d097" containerID="83360a584c931667c0b5cc0b37c0264138ea12b4ecf92cdc432c991b78793937" exitCode=60
Nov 01 07:00:18.244382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.244290    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerDied","Data":"83360a584c931667c0b5cc0b37c0264138ea12b4ecf92cdc432c991b78793937"}
Nov 01 07:00:18.244720 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.244704    4468 scope.go:122] "RemoveContainer" containerID="83360a584c931667c0b5cc0b37c0264138ea12b4ecf92cdc432c991b78793937"
Nov 01 07:00:18.244980 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:18.244960    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx_odh-ai-gateway-infra(a8787183-583d-4e63-be9d-8b211b87d097)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" podUID="a8787183-583d-4e63-be9d-8b211b87d097"
Nov 01 07:00:18.246240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.246222    4468 generic.go:356] "Generic (PLEG): container finished" podID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerID="554841df0ac7b746928d0a01cc156ab227e60f644258b22396d80a6ee12e3752" exitCode=60
Nov 01 07:00:18.246352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.246266    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerDied","Data":"554841df0ac7b746928d0a01cc156ab227e60f644258b22396d80a6ee12e3752"}
Nov 01 07:00:18.246534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.246519    4468 scope.go:122] "RemoveContainer" containerID="554841df0ac7b746928d0a01cc156ab227e60f644258b22396d80a6ee12e3752"
Nov 01 07:00:18.246763 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:18.246739    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms_odh-ai-gateway-infra(7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82"
Nov 01 07:00:18.252261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.252243    4468 scope.go:122] "RemoveContainer" containerID="fb6787d60ab0c8d9a5ebaf4d02f5db5cd1f29b331831ac26123ac217947e3522"
Nov 01 07:00:18.264993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.264973    4468 scope.go:122] "RemoveContainer" containerID="69528c35e6f4f3ef560a1663f2208cc8041688326bac4f6acb9d602ad429daa2"
Nov 01 07:00:18.376058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.376036    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:18.464784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.464716    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca\") pod \"be39e287-82ea-4425-8b81-4b30381675d5\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") "
Nov 01 07:00:18.464924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.464795    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8\" (UniqueName: \"kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8\") pod \"be39e287-82ea-4425-8b81-4b30381675d5\" (UID: \"be39e287-82ea-4425-8b81-4b30381675d5\") "
Nov 01 07:00:18.465039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.465018    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca" pod "be39e287-82ea-4425-8b81-4b30381675d5" (UID: "be39e287-82ea-4425-8b81-4b30381675d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:18.465414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.465398    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be39e287-82ea-4425-8b81-4b30381675d5-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:18.465506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.465484    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"]
Nov 01 07:00:18.467348 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.467319    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8" pod "be39e287-82ea-4425-8b81-4b30381675d5" (UID: "be39e287-82ea-4425-8b81-4b30381675d5"). InnerVolumeSpecName "kube-api-access-hnph8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:18.470657 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.470616    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2" podStartSLOduration=2.470604217 podStartE2EDuration="2.470604217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:18.467552554 +0000 UTC m=+3030.035359386" watchObservedRunningTime="2026-11-01 07:00:18.470604217 +0000 UTC m=+3030.038411045"
Nov 01 07:00:18.566570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.566534    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnph8\" (UniqueName: \"kubernetes.io/projected/be39e287-82ea-4425-8b81-4b30381675d5-kube-api-access-hnph8\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:18.575555 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.575532    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"]
Nov 01 07:00:18.583450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.583433    4468 scope.go:122] "RemoveContainer" containerID="20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a"
Nov 01 07:00:18.583591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.583573    4468 scope.go:122] "RemoveContainer" containerID="9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0"
Nov 01 07:00:18.583791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.583772    4468 scope.go:122] "RemoveContainer" containerID="2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd"
Nov 01 07:00:18.690692 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:18.690667    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"]
Nov 01 07:00:19.259203 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.259172    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerStarted","Data":"04587e75f5bd982a6e909b43f1f2362fa057f84c0737e94dfb559b5427c14285"}
Nov 01 07:00:19.261240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.261211    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerStarted","Data":"041329dc5b5a58c2f188b5eeaa8c25e7b700ac79ed068a0c67c20e47fea848ad"}
Nov 01 07:00:19.263182 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.263159    4468 generic.go:356] "Generic (PLEG): container finished" podID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerID="832dafa22bfa3a93037cd0ed56fc7f0f05e3efdb0cb935ce7799a5f240eac6a8" exitCode=60
Nov 01 07:00:19.263270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.263239    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerDied","Data":"832dafa22bfa3a93037cd0ed56fc7f0f05e3efdb0cb935ce7799a5f240eac6a8"}
Nov 01 07:00:19.263346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.263299    4468 scope.go:122] "RemoveContainer" containerID="c2ec8f2d194a5949e84330a8d01acab19d9d1d798426549ef1fd6023f0b492ef"
Nov 01 07:00:19.263743 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.263725    4468 scope.go:122] "RemoveContainer" containerID="832dafa22bfa3a93037cd0ed56fc7f0f05e3efdb0cb935ce7799a5f240eac6a8"
Nov 01 07:00:19.264122 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:19.264081    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc_odh-ai-gateway-infra(4e690403-5f4d-479e-8a60-403ed4cb1cf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6"
Nov 01 07:00:19.265857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.265827    4468 generic.go:356] "Generic (PLEG): container finished" podID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerID="4bfee6d05e3f94a808e6c84163f22909ee70c5dbcf0b89290915767b1b13fb6c" exitCode=60
Nov 01 07:00:19.265975 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.265938    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerDied","Data":"4bfee6d05e3f94a808e6c84163f22909ee70c5dbcf0b89290915767b1b13fb6c"}
Nov 01 07:00:19.266347 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.266330    4468 scope.go:122] "RemoveContainer" containerID="4bfee6d05e3f94a808e6c84163f22909ee70c5dbcf0b89290915767b1b13fb6c"
Nov 01 07:00:19.266659 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:19.266636    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv_odh-ai-gateway-infra(717f3bb7-a336-4920-ac05-9c722a5bf9af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af"
Nov 01 07:00:19.268570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.268548    4468 generic.go:356] "Generic (PLEG): container finished" podID="ae53244a-80f1-4f4b-823a-dad36025b759" containerID="ef588fd6cfa974bd0d8fb95c57e2e21a24dc6030f16998059ef6ab2b5e1d91e8" exitCode=0
Nov 01 07:00:19.268674 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.268601    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2" event={"ID":"ae53244a-80f1-4f4b-823a-dad36025b759","Type":"ContainerDied","Data":"ef588fd6cfa974bd0d8fb95c57e2e21a24dc6030f16998059ef6ab2b5e1d91e8"}
Nov 01 07:00:19.272200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.272181    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"
Nov 01 07:00:19.272842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.272823    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch" event={"ID":"be39e287-82ea-4425-8b81-4b30381675d5","Type":"ContainerDied","Data":"46402429170d3fadeccfa1f92bc64daca11300c02913bbdd709ff0146a428450"}
Nov 01 07:00:19.276690 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.275444    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerStarted","Data":"993acbcfbc55837226c80c9d640d93656def7a7360b9669e137350f65675c1ae"}
Nov 01 07:00:19.280067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.279118    4468 generic.go:356] "Generic (PLEG): container finished" podID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerID="69cb69cd1dfba0e16ce2b6398a053fae70cba766b727b53936738af443e7746b" exitCode=60
Nov 01 07:00:19.280067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.279187    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerDied","Data":"69cb69cd1dfba0e16ce2b6398a053fae70cba766b727b53936738af443e7746b"}
Nov 01 07:00:19.280067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.279505    4468 scope.go:122] "RemoveContainer" containerID="69cb69cd1dfba0e16ce2b6398a053fae70cba766b727b53936738af443e7746b"
Nov 01 07:00:19.280067 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:19.279757    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42_odh-ai-gateway-infra(dfe256c8-4251-4a0f-9c7e-786bc5954f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21"
Nov 01 07:00:19.282925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.282166    4468 generic.go:356] "Generic (PLEG): container finished" podID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerID="bdfc1833b17ab2ff9a838b92d1e63bb9f635f47b3db4f17b37d64e9d384983eb" exitCode=60
Nov 01 07:00:19.282925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.282210    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerDied","Data":"bdfc1833b17ab2ff9a838b92d1e63bb9f635f47b3db4f17b37d64e9d384983eb"}
Nov 01 07:00:19.282925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.282554    4468 scope.go:122] "RemoveContainer" containerID="bdfc1833b17ab2ff9a838b92d1e63bb9f635f47b3db4f17b37d64e9d384983eb"
Nov 01 07:00:19.282925 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:19.282827    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d_odh-ai-gateway-infra(b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7"
Nov 01 07:00:19.323522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.311641    4468 scope.go:122] "RemoveContainer" containerID="0fe6a85aab930a374e96943e0c872872c66e03c0c37206f254bd60741daafbcd"
Nov 01 07:00:19.364664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.364083    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"]
Nov 01 07:00:19.368875 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.368848    4468 scope.go:122] "RemoveContainer" containerID="4e81ad1e4aaaa63413cf45c53fc6afdae69d183145b6d1bf384370f8d665f129"
Nov 01 07:00:19.388641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.388616    4468 scope.go:122] "RemoveContainer" containerID="5d159b93633f78edc231f8242c22681911307223773b2e29ae7772223da5a6d7"
Nov 01 07:00:19.402256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.402234    4468 scope.go:122] "RemoveContainer" containerID="419400adb24cb013dfdcf1550684d20876c9b082d66988e80d9cfd596cc7ee9c"
Nov 01 07:00:19.443616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.443554    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:19.529181 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.529154    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:19.532627 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.532605    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:19.542265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.542242    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"]
Nov 01 07:00:19.545629 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.545603    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-28c598-29891940-kblch"]
Nov 01 07:00:19.582019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.581996    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m\" (UniqueName: \"kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m\") pod \"a8787183-583d-4e63-be9d-8b211b87d097\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") "
Nov 01 07:00:19.582240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.582212    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca\") pod \"a8787183-583d-4e63-be9d-8b211b87d097\" (UID: \"a8787183-583d-4e63-be9d-8b211b87d097\") "
Nov 01 07:00:19.582630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.582605    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca" pod "a8787183-583d-4e63-be9d-8b211b87d097" (UID: "a8787183-583d-4e63-be9d-8b211b87d097"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:19.583564 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.583548    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8787183-583d-4e63-be9d-8b211b87d097-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.584576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.584560    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m" pod "a8787183-583d-4e63-be9d-8b211b87d097" (UID: "a8787183-583d-4e63-be9d-8b211b87d097"). InnerVolumeSpecName "kube-api-access-5rx5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:19.684225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.684189    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk\" (UniqueName: \"kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk\") pod \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") "
Nov 01 07:00:19.684420 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.684378    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca\") pod \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\" (UID: \"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82\") "
Nov 01 07:00:19.684530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.684441    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc\" (UniqueName: \"kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc\") pod \"a1015013-8668-4ff1-8268-065840ae0a60\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") "
Nov 01 07:00:19.684530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.684488    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca\") pod \"a1015013-8668-4ff1-8268-065840ae0a60\" (UID: \"a1015013-8668-4ff1-8268-065840ae0a60\") "
Nov 01 07:00:19.684784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.684759    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca" pod "7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" (UID: "7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:19.684867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.684830    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca" pod "a1015013-8668-4ff1-8268-065840ae0a60" (UID: "a1015013-8668-4ff1-8268-065840ae0a60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:19.685130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.685114    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rx5m\" (UniqueName: \"kubernetes.io/projected/a8787183-583d-4e63-be9d-8b211b87d097-kube-api-access-5rx5m\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.685192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.685133    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.685192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.685142    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1015013-8668-4ff1-8268-065840ae0a60-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.686582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.686558    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk" pod "7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" (UID: "7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82"). InnerVolumeSpecName "kube-api-access-66pkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:19.686773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.686756    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc" pod "a1015013-8668-4ff1-8268-065840ae0a60" (UID: "a1015013-8668-4ff1-8268-065840ae0a60"). InnerVolumeSpecName "kube-api-access-tb9tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:19.785995 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.785957    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pkk\" (UniqueName: \"kubernetes.io/projected/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82-kube-api-access-66pkk\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.785995 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.785988    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb9tc\" (UniqueName: \"kubernetes.io/projected/a1015013-8668-4ff1-8268-065840ae0a60-kube-api-access-tb9tc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:19.846153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:19.846125    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"]
Nov 01 07:00:20.181982 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.181904    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"]
Nov 01 07:00:20.294620 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.294591    4468 generic.go:356] "Generic (PLEG): container finished" podID="32836548-dfd4-4309-8897-095ab10d6909" containerID="993acbcfbc55837226c80c9d640d93656def7a7360b9669e137350f65675c1ae" exitCode=60
Nov 01 07:00:20.295065 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.294667    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerDied","Data":"993acbcfbc55837226c80c9d640d93656def7a7360b9669e137350f65675c1ae"}
Nov 01 07:00:20.295065 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.294723    4468 scope.go:122] "RemoveContainer" containerID="2ee014208a021a94465b4d5e1c2da421e18fc22a21371841fa29664bcc9cc5cd"
Nov 01 07:00:20.295183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.295081    4468 scope.go:122] "RemoveContainer" containerID="993acbcfbc55837226c80c9d640d93656def7a7360b9669e137350f65675c1ae"
Nov 01 07:00:20.295419 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:20.295394    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56_odh-ai-gateway-infra(32836548-dfd4-4309-8897-095ab10d6909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" podUID="32836548-dfd4-4309-8897-095ab10d6909"
Nov 01 07:00:20.299363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.299339    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx" event={"ID":"a8787183-583d-4e63-be9d-8b211b87d097","Type":"ContainerDied","Data":"7a2d88e8e155f6cb5b2ea3ec26d1b1e718856b4f2b0fb07cc5b2313667c2e2ce"}
Nov 01 07:00:20.299363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.299358    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"
Nov 01 07:00:20.300842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.300821    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms" event={"ID":"7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82","Type":"ContainerDied","Data":"f7a99dbc488b778564c66f98c8ca796a7a47c06f4d8f44b513974f2de0ec0fed"}
Nov 01 07:00:20.300842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.300837    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"
Nov 01 07:00:20.304061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.304035    4468 generic.go:356] "Generic (PLEG): container finished" podID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerID="04587e75f5bd982a6e909b43f1f2362fa057f84c0737e94dfb559b5427c14285" exitCode=60
Nov 01 07:00:20.304163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.304147    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerDied","Data":"04587e75f5bd982a6e909b43f1f2362fa057f84c0737e94dfb559b5427c14285"}
Nov 01 07:00:20.304579 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.304558    4468 scope.go:122] "RemoveContainer" containerID="04587e75f5bd982a6e909b43f1f2362fa057f84c0737e94dfb559b5427c14285"
Nov 01 07:00:20.304874 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:20.304849    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl_odh-ai-gateway-infra(1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6"
Nov 01 07:00:20.306653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.306628    4468 generic.go:356] "Generic (PLEG): container finished" podID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerID="041329dc5b5a58c2f188b5eeaa8c25e7b700ac79ed068a0c67c20e47fea848ad" exitCode=60
Nov 01 07:00:20.306763 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.306705    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerDied","Data":"041329dc5b5a58c2f188b5eeaa8c25e7b700ac79ed068a0c67c20e47fea848ad"}
Nov 01 07:00:20.307069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.307050    4468 scope.go:122] "RemoveContainer" containerID="041329dc5b5a58c2f188b5eeaa8c25e7b700ac79ed068a0c67c20e47fea848ad"
Nov 01 07:00:20.307376 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:20.307351    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891940-rtsnc_odh-ai-gateway-infra(5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3"
Nov 01 07:00:20.310416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.310226    4468 scope.go:122] "RemoveContainer" containerID="83360a584c931667c0b5cc0b37c0264138ea12b4ecf92cdc432c991b78793937"
Nov 01 07:00:20.315107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.315086    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw" event={"ID":"a1015013-8668-4ff1-8268-065840ae0a60","Type":"ContainerDied","Data":"d854f8f21070b7bdcf3bd10323c171cd1fd2a87197f577c2ba84ddc86a777bb1"}
Nov 01 07:00:20.315255 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.315186    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"
Nov 01 07:00:20.332621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.332588    4468 scope.go:122] "RemoveContainer" containerID="554841df0ac7b746928d0a01cc156ab227e60f644258b22396d80a6ee12e3752"
Nov 01 07:00:20.350414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.350389    4468 scope.go:122] "RemoveContainer" containerID="20fe1000f67186dea58473eae4449574188054e6ed109c12e1247d3ff2bbe21a"
Nov 01 07:00:20.373179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.373153    4468 scope.go:122] "RemoveContainer" containerID="9bd01c31ed9e8c24548f9b2ba21937979902480307a044395c39fb03d90107e0"
Nov 01 07:00:20.403668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.403303    4468 scope.go:122] "RemoveContainer" containerID="f29898ed5df7a386a788d23cf6b08adadced37979c1148e897a8bb6b221054af"
Nov 01 07:00:20.478755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.478725    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:20.570704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.570682    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:20.574215 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.574197    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:20.578037 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.578018    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:20.591640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.591615    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be39e287-82ea-4425-8b81-4b30381675d5" path="/var/lib/kubelet/pods/be39e287-82ea-4425-8b81-4b30381675d5/volumes"
Nov 01 07:00:20.595311 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.595288    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca\") pod \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") "
Nov 01 07:00:20.595513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.595497    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2\" (UniqueName: \"kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2\") pod \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\" (UID: \"dfe256c8-4251-4a0f-9c7e-786bc5954f21\") "
Nov 01 07:00:20.595623 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.595602    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca" pod "dfe256c8-4251-4a0f-9c7e-786bc5954f21" (UID: "dfe256c8-4251-4a0f-9c7e-786bc5954f21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:20.595949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.595930    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe256c8-4251-4a0f-9c7e-786bc5954f21-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.598143 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.598126    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2" pod "dfe256c8-4251-4a0f-9c7e-786bc5954f21" (UID: "dfe256c8-4251-4a0f-9c7e-786bc5954f21"). InnerVolumeSpecName "kube-api-access-htnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:20.697071 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697042    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9\" (UniqueName: \"kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9\") pod \"717f3bb7-a336-4920-ac05-9c722a5bf9af\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") "
Nov 01 07:00:20.697227 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697218    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca\") pod \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") "
Nov 01 07:00:20.697313 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697238    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q\" (UniqueName: \"kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q\") pod \"ae53244a-80f1-4f4b-823a-dad36025b759\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") "
Nov 01 07:00:20.697313 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697260    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca\") pod \"717f3bb7-a336-4920-ac05-9c722a5bf9af\" (UID: \"717f3bb7-a336-4920-ac05-9c722a5bf9af\") "
Nov 01 07:00:20.697423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697314    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca\") pod \"ae53244a-80f1-4f4b-823a-dad36025b759\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") "
Nov 01 07:00:20.697423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697374    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg\" (UniqueName: \"kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg\") pod \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\" (UID: \"4e690403-5f4d-479e-8a60-403ed4cb1cf6\") "
Nov 01 07:00:20.697423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697402    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca\") pod \"ae53244a-80f1-4f4b-823a-dad36025b759\" (UID: \"ae53244a-80f1-4f4b-823a-dad36025b759\") "
Nov 01 07:00:20.697665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697635    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca" pod "717f3bb7-a336-4920-ac05-9c722a5bf9af" (UID: "717f3bb7-a336-4920-ac05-9c722a5bf9af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:20.697749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca" pod "4e690403-5f4d-479e-8a60-403ed4cb1cf6" (UID: "4e690403-5f4d-479e-8a60-403ed4cb1cf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:20.697982 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.697954    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca" pod "ae53244a-80f1-4f4b-823a-dad36025b759" (UID: "ae53244a-80f1-4f4b-823a-dad36025b759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:20.698124 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.698104    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e690403-5f4d-479e-8a60-403ed4cb1cf6-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.698202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.698131    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/717f3bb7-a336-4920-ac05-9c722a5bf9af-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.698202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.698146    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.698202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.698162    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnl2\" (UniqueName: \"kubernetes.io/projected/dfe256c8-4251-4a0f-9c7e-786bc5954f21-kube-api-access-htnl2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.699495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.699452    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9" pod "717f3bb7-a336-4920-ac05-9c722a5bf9af" (UID: "717f3bb7-a336-4920-ac05-9c722a5bf9af"). InnerVolumeSpecName "kube-api-access-7hgz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:20.699661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.699639    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg" pod "4e690403-5f4d-479e-8a60-403ed4cb1cf6" (UID: "4e690403-5f4d-479e-8a60-403ed4cb1cf6"). InnerVolumeSpecName "kube-api-access-47fkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:20.699759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.699720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q" pod "ae53244a-80f1-4f4b-823a-dad36025b759" (UID: "ae53244a-80f1-4f4b-823a-dad36025b759"). InnerVolumeSpecName "kube-api-access-67z8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:20.699930 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.699908    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca" pod "ae53244a-80f1-4f4b-823a-dad36025b759" (UID: "ae53244a-80f1-4f4b-823a-dad36025b759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:00:20.799031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.799008    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae53244a-80f1-4f4b-823a-dad36025b759-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.799031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.799031    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hgz9\" (UniqueName: \"kubernetes.io/projected/717f3bb7-a336-4920-ac05-9c722a5bf9af-kube-api-access-7hgz9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.799207 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.799041    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67z8q\" (UniqueName: \"kubernetes.io/projected/ae53244a-80f1-4f4b-823a-dad36025b759-kube-api-access-67z8q\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.799207 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.799051    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fkg\" (UniqueName: \"kubernetes.io/projected/4e690403-5f4d-479e-8a60-403ed4cb1cf6-kube-api-access-47fkg\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:20.886783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:20.886753    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"]
Nov 01 07:00:21.067047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.066970    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"]
Nov 01 07:00:21.070360 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.070335    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ff1dc-29891940-txcms"]
Nov 01 07:00:21.111121 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.111089    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"]
Nov 01 07:00:21.223906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.223873    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"]
Nov 01 07:00:21.330647 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.330569    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"
Nov 01 07:00:21.330647 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.330599    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42" event={"ID":"dfe256c8-4251-4a0f-9c7e-786bc5954f21","Type":"ContainerDied","Data":"d8f108be109e89334bf01bdc99b95e2e7ff3b2510c1e70184657b062845bba3d"}
Nov 01 07:00:21.332876 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.330660    4468 scope.go:122] "RemoveContainer" containerID="69cb69cd1dfba0e16ce2b6398a053fae70cba766b727b53936738af443e7746b"
Nov 01 07:00:21.338915 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.338887    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"]
Nov 01 07:00:21.348806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.347976    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc" event={"ID":"4e690403-5f4d-479e-8a60-403ed4cb1cf6","Type":"ContainerDied","Data":"8f8c3c39b41bf360c8e9e6a3c70fac2bd28b1b2d900a95977b2e9586bea6c21c"}
Nov 01 07:00:21.348806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.348062    4468 scope.go:122] "RemoveContainer" containerID="832dafa22bfa3a93037cd0ed56fc7f0f05e3efdb0cb935ce7799a5f240eac6a8"
Nov 01 07:00:21.348806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.348143    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"
Nov 01 07:00:21.357277 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.357239    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv" event={"ID":"717f3bb7-a336-4920-ac05-9c722a5bf9af","Type":"ContainerDied","Data":"34e30c53a64a3dfb1ec5486235366ec5afe17c6d720cafbb3130ef8113eacab7"}
Nov 01 07:00:21.357509 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.357366    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"
Nov 01 07:00:21.373224 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.373200    4468 scope.go:122] "RemoveContainer" containerID="4bfee6d05e3f94a808e6c84163f22909ee70c5dbcf0b89290915767b1b13fb6c"
Nov 01 07:00:21.378105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.377000    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"
Nov 01 07:00:21.378105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.377540    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2" event={"ID":"ae53244a-80f1-4f4b-823a-dad36025b759","Type":"ContainerDied","Data":"4d60836423885b8b7cf7a44553013ddde2274ab76ff9b81138bcc824481b2708"}
Nov 01 07:00:21.378105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.377596    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d60836423885b8b7cf7a44553013ddde2274ab76ff9b81138bcc824481b2708"
Nov 01 07:00:21.591630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.589874    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:21.602387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.602362    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"]
Nov 01 07:00:21.605829 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.605807    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:21.607542 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.607518    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c831a4c0-29891940-nv8cw"]
Nov 01 07:00:21.610686 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.610668    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:21.668404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.668381    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:21.716591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716563    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca\") pod \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") "
Nov 01 07:00:21.716741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716603    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb\" (UniqueName: \"kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb\") pod \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") "
Nov 01 07:00:21.716741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716695    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca\") pod \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\" (UID: \"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7\") "
Nov 01 07:00:21.716824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716741    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6\" (UniqueName: \"kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6\") pod \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") "
Nov 01 07:00:21.716824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716772    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt\" (UniqueName: \"kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt\") pod \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\" (UID: \"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6\") "
Nov 01 07:00:21.716982 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716828    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca\") pod \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\" (UID: \"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3\") "
Nov 01 07:00:21.717044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.716978    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca" pod "1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" (UID: "1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:21.717103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.717057    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca" pod "b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" (UID: "b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:21.717296 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.717257    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca" pod "5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" (UID: "5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:21.717825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.717803    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.717825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.717832    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.717986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.717848    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.719073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.719049    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6" pod "5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" (UID: "5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3"). InnerVolumeSpecName "kube-api-access-wzmr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:21.719167 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.719085    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb" pod "b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" (UID: "b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7"). InnerVolumeSpecName "kube-api-access-qcvjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:21.719212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.719164    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt" pod "1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" (UID: "1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6"). InnerVolumeSpecName "kube-api-access-ts4mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:21.819017 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.818962    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt\" (UniqueName: \"kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt\") pod \"32836548-dfd4-4309-8897-095ab10d6909\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") "
Nov 01 07:00:21.819179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.819073    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca\") pod \"32836548-dfd4-4309-8897-095ab10d6909\" (UID: \"32836548-dfd4-4309-8897-095ab10d6909\") "
Nov 01 07:00:21.819567 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.819539    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvjb\" (UniqueName: \"kubernetes.io/projected/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7-kube-api-access-qcvjb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.819697 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.819569    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzmr6\" (UniqueName: \"kubernetes.io/projected/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3-kube-api-access-wzmr6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.819697 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.819583    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4mt\" (UniqueName: \"kubernetes.io/projected/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6-kube-api-access-ts4mt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.820067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.820024    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca" pod "32836548-dfd4-4309-8897-095ab10d6909" (UID: "32836548-dfd4-4309-8897-095ab10d6909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:21.821771 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.821748    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt" pod "32836548-dfd4-4309-8897-095ab10d6909" (UID: "32836548-dfd4-4309-8897-095ab10d6909"). InnerVolumeSpecName "kube-api-access-mdjqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:21.921328 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.921248    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdjqt\" (UniqueName: \"kubernetes.io/projected/32836548-dfd4-4309-8897-095ab10d6909-kube-api-access-mdjqt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.921328 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.921285    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32836548-dfd4-4309-8897-095ab10d6909-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:21.982151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.982119    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"]
Nov 01 07:00:21.985621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:21.985598    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-581487-29891940-6x5lx"]
Nov 01 07:00:22.098287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.098256    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"]
Nov 01 07:00:22.101552 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.101526    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-302ba0-29891940-7xd42"]
Nov 01 07:00:22.406268 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.406241    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"
Nov 01 07:00:22.406268 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.406265    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56" event={"ID":"32836548-dfd4-4309-8897-095ab10d6909","Type":"ContainerDied","Data":"52672370b8f8e3af5aa153b4246464beddda69d45a72808d3177d6301d6f8e17"}
Nov 01 07:00:22.406862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.406333    4468 scope.go:122] "RemoveContainer" containerID="993acbcfbc55837226c80c9d640d93656def7a7360b9669e137350f65675c1ae"
Nov 01 07:00:22.409874 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.409849    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"
Nov 01 07:00:22.413626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.413604    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d" event={"ID":"b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7","Type":"ContainerDied","Data":"cb6afe30be8dbf44fec0890c2c5d001cc7d688c101ef553e9cd342cc3b8aa76c"}
Nov 01 07:00:22.414969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.414947    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl" event={"ID":"1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6","Type":"ContainerDied","Data":"f1140f6395c877d40cfc932e0a4c97b55be422e4771707c4759bad778920be2f"}
Nov 01 07:00:22.415070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.415031    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"
Nov 01 07:00:22.417658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.417642    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"
Nov 01 07:00:22.417890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.417646    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc" event={"ID":"5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3","Type":"ContainerDied","Data":"b459168707c826dfba61c4866a971890f819c8ecbec0e76facf9b4e14a944e45"}
Nov 01 07:00:22.421518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.421497    4468 scope.go:122] "RemoveContainer" containerID="bdfc1833b17ab2ff9a838b92d1e63bb9f635f47b3db4f17b37d64e9d384983eb"
Nov 01 07:00:22.432929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.432913    4468 scope.go:122] "RemoveContainer" containerID="04587e75f5bd982a6e909b43f1f2362fa057f84c0737e94dfb559b5427c14285"
Nov 01 07:00:22.441899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.441882    4468 scope.go:122] "RemoveContainer" containerID="041329dc5b5a58c2f188b5eeaa8c25e7b700ac79ed068a0c67c20e47fea848ad"
Nov 01 07:00:22.591025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.590990    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" path="/var/lib/kubelet/pods/7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82/volumes"
Nov 01 07:00:22.591494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.591476    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1015013-8668-4ff1-8268-065840ae0a60" path="/var/lib/kubelet/pods/a1015013-8668-4ff1-8268-065840ae0a60/volumes"
Nov 01 07:00:22.591897 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.591884    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8787183-583d-4e63-be9d-8b211b87d097" path="/var/lib/kubelet/pods/a8787183-583d-4e63-be9d-8b211b87d097/volumes"
Nov 01 07:00:22.592888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.592751    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" path="/var/lib/kubelet/pods/dfe256c8-4251-4a0f-9c7e-786bc5954f21/volumes"
Nov 01 07:00:22.705282 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.705191    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"]
Nov 01 07:00:22.708774 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.708750    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-3db672-29891940-bdqqv"]
Nov 01 07:00:22.816548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.816521    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"]
Nov 01 07:00:22.822273 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.822251    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891940-j79gc"]
Nov 01 07:00:22.929910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.929879    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"]
Nov 01 07:00:22.934385 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:22.934362    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-51aa8f-29891940-r5v56"]
Nov 01 07:00:23.045548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:23.045529    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"]
Nov 01 07:00:23.051392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:23.051364    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25719d-29891940-ph8tl"]
Nov 01 07:00:23.158909 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:23.158755    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"]
Nov 01 07:00:23.162101 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:23.162079    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891940-rtsnc"]
Nov 01 07:00:23.269363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:23.269334    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"]
Nov 01 07:00:23.274508 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:23.274488    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891940-w267d"]
Nov 01 07:00:24.597520 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:24.597456    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" path="/var/lib/kubelet/pods/1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6/volumes"
Nov 01 07:00:24.597934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:24.597919    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32836548-dfd4-4309-8897-095ab10d6909" path="/var/lib/kubelet/pods/32836548-dfd4-4309-8897-095ab10d6909/volumes"
Nov 01 07:00:24.598382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:24.598365    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" path="/var/lib/kubelet/pods/4e690403-5f4d-479e-8a60-403ed4cb1cf6/volumes"
Nov 01 07:00:24.599050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:24.599034    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" path="/var/lib/kubelet/pods/5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3/volumes"
Nov 01 07:00:24.599390 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:24.599376    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" path="/var/lib/kubelet/pods/717f3bb7-a336-4920-ac05-9c722a5bf9af/volumes"
Nov 01 07:00:24.599877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:24.599859    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" path="/var/lib/kubelet/pods/b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7/volumes"
Nov 01 07:00:27.025457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.025425    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"]
Nov 01 07:00:27.026001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.025987    4468 cpu_manager.go:414] "containerMap: removing container" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.026041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026003    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.026041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026023    4468 cpu_manager.go:414] "containerMap: removing container" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.026041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026027    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.026041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026035    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.026041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026040    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026047    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026051    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026060    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026064    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026071    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026076    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026083    4468 cpu_manager.go:414] "containerMap: removing container" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026087    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026092    4468 cpu_manager.go:414] "containerMap: removing container" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026096    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026103    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026107    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026113    4468 cpu_manager.go:414] "containerMap: removing container" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026117    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026125    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026129    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026137    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026141    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026151    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026154    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026161    4468 cpu_manager.go:414] "containerMap: removing container" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026166    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026171    4468 cpu_manager.go:414] "containerMap: removing container" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026175    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026182    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026185    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026192    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026196    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026202    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:27.026197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026206    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026213    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026217    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026221    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026225    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026230    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ae53244a-80f1-4f4b-823a-dad36025b759" containerName="revoke-keys"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026234    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae53244a-80f1-4f4b-823a-dad36025b759" containerName="revoke-keys"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026241    4468 cpu_manager.go:414] "containerMap: removing container" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026244    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026250    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026253    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026262    4468 cpu_manager.go:414] "containerMap: removing container" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026266    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026275    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026279    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026287    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026291    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026296    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026300    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026306    4468 cpu_manager.go:414] "containerMap: removing container" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026309    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026386    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026393    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026398    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026405    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026411    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026416    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026421    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026427    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026431    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae53244a-80f1-4f4b-823a-dad36025b759" containerName="revoke-keys"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026437    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026442    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026447    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026452    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026459    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026481    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026488    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026496    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026504    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026511    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026516    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026520    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026526    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026531    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026544    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026551    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026557    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="be39e287-82ea-4425-8b81-4b30381675d5" containerName="cleanup"
Nov 01 07:00:27.027058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.026564    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:27.031493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.031453    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.087542 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.086514    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"]
Nov 01 07:00:27.131606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.131575    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0cb4ef\"/\"explicit-model-0cb4ef-kserve-self-signed-certs\""
Nov 01 07:00:27.133105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.133082    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0cb4ef\"/\"kube-root-ca.crt\""
Nov 01 07:00:27.133265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.133225    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0cb4ef\"/\"openshift-service-ca.crt\""
Nov 01 07:00:27.133265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.133248    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0cb4ef\"/\"default-dockercfg-56wjd\""
Nov 01 07:00:27.212497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.211658    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.212497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.211794    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8wc\" (UniqueName: \"kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.212497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.211833    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.212497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.211868    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.212497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.211913    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.212497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.211936    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314038 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.313993    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8wc\" (UniqueName: \"kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314051    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314084    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314127    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314209    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314442    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314533 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314478    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314533 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314498    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.314722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.314702    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.316626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.316608    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.316921 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.316897    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.380026 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.379995    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8wc\" (UniqueName: \"kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc\") pod \"explicit-model-0cb4ef-kserve-7c546f5479-54btl\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") " pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:27.643644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:27.643554    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:28.040374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.040346    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"]
Nov 01 07:00:28.041061 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:28.041032    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd26d1d65_0b4a_4484_bcef_32c77f491dd5.slice/crio-a148353f161d41196d921a4b6aed14d9c4079a8ec8e9c1e413113adadddecbfc WatchSource:0}: Error finding container a148353f161d41196d921a4b6aed14d9c4079a8ec8e9c1e413113adadddecbfc: Status 404 returned error can't find the container with id a148353f161d41196d921a4b6aed14d9c4079a8ec8e9c1e413113adadddecbfc
Nov 01 07:00:28.466958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.466918    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl" event={"ID":"d26d1d65-0b4a-4484-bcef-32c77f491dd5","Type":"ContainerStarted","Data":"b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604"}
Nov 01 07:00:28.466958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.466965    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl" event={"ID":"d26d1d65-0b4a-4484-bcef-32c77f491dd5","Type":"ContainerStarted","Data":"a148353f161d41196d921a4b6aed14d9c4079a8ec8e9c1e413113adadddecbfc"}
Nov 01 07:00:28.584099 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.584055    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl" podStartSLOduration=2.584041814 podStartE2EDuration="2.584041814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:28.581622508 +0000 UTC m=+3040.149429370" watchObservedRunningTime="2026-11-01 07:00:28.584041814 +0000 UTC m=+3040.151848643"
Nov 01 07:00:28.634319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634290    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"]
Nov 01 07:00:28.634881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634866    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634884    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddbd24a-5ab5-42b7-bbfd-dc71a0f8bf82" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634894    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634899    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634916    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634921    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634936    4468 cpu_manager.go:414] "containerMap: removing container" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:28.634946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634940    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe256c8-4251-4a0f-9c7e-786bc5954f21" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634957    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634962    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd8eaf6-11f3-40ad-aa1c-2e7ecb3f05c6" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634972    4468 cpu_manager.go:414] "containerMap: removing container" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.634976    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635054    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2b8ad1-fc50-4904-89c5-5e801e0fd0a3" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635063    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="717f3bb7-a336-4920-ac05-9c722a5bf9af" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635071    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="32836548-dfd4-4309-8897-095ab10d6909" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635078    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e690403-5f4d-479e-8a60-403ed4cb1cf6" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635083    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0de880d-b6f4-45e4-b66e-1acd9c7ec1d7" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635091    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8787183-583d-4e63-be9d-8b211b87d097" containerName="cleanup"
Nov 01 07:00:28.635152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.635099    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1015013-8668-4ff1-8268-065840ae0a60" containerName="cleanup"
Nov 01 07:00:28.638541 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.638517    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.644532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.644510    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:28.690807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.690781    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:00:28.690944 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.690911    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:00:28.690944 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.690911    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:00:28.691049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.690952    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:00:28.711450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.711412    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"]
Nov 01 07:00:28.731334 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.731253    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.731522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.731500    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.731585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.731531    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxc4\" (UniqueName: \"kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.832882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.832851    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.832882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.832885    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxc4\" (UniqueName: \"kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.833070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.832986    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.833457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.833438    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.835559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.835538    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.888878 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.888851    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxc4\" (UniqueName: \"kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4\") pod \"maas-api-revoke-sub-78a2a501-vkp5r\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:28.951731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:28.951706    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:29.322561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:29.322526    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"]
Nov 01 07:00:29.323042 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:29.323013    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ef186c_6f92_4602_90f7_e1e472229adf.slice/crio-f6e3c3507c6e4e66f9108944d1fc8b4ae19de49ef6891b26f62bfc9af6b461c7 WatchSource:0}: Error finding container f6e3c3507c6e4e66f9108944d1fc8b4ae19de49ef6891b26f62bfc9af6b461c7: Status 404 returned error can't find the container with id f6e3c3507c6e4e66f9108944d1fc8b4ae19de49ef6891b26f62bfc9af6b461c7
Nov 01 07:00:29.486098 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:29.486047    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r" event={"ID":"e4ef186c-6f92-4602-90f7-e1e472229adf","Type":"ContainerStarted","Data":"f6e3c3507c6e4e66f9108944d1fc8b4ae19de49ef6891b26f62bfc9af6b461c7"}
Nov 01 07:00:30.494983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:30.494902    4468 generic.go:356] "Generic (PLEG): container finished" podID="e4ef186c-6f92-4602-90f7-e1e472229adf" containerID="d382b4a953894c0539366851006c22a42bdcd60a8f28c8c0fcc590f4ef3b9bd6" exitCode=0
Nov 01 07:00:30.495357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:30.494983    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r" event={"ID":"e4ef186c-6f92-4602-90f7-e1e472229adf","Type":"ContainerDied","Data":"d382b4a953894c0539366851006c22a42bdcd60a8f28c8c0fcc590f4ef3b9bd6"}
Nov 01 07:00:31.645916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.645893    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:31.667386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.667359    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca\") pod \"e4ef186c-6f92-4602-90f7-e1e472229adf\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") "
Nov 01 07:00:31.667518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.667420    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4\" (UniqueName: \"kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4\") pod \"e4ef186c-6f92-4602-90f7-e1e472229adf\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") "
Nov 01 07:00:31.667634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.667619    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca\") pod \"e4ef186c-6f92-4602-90f7-e1e472229adf\" (UID: \"e4ef186c-6f92-4602-90f7-e1e472229adf\") "
Nov 01 07:00:31.667968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.667945    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca" pod "e4ef186c-6f92-4602-90f7-e1e472229adf" (UID: "e4ef186c-6f92-4602-90f7-e1e472229adf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:31.668378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.668357    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:31.669435 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.669410    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca" pod "e4ef186c-6f92-4602-90f7-e1e472229adf" (UID: "e4ef186c-6f92-4602-90f7-e1e472229adf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:00:31.669563 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.669501    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4" pod "e4ef186c-6f92-4602-90f7-e1e472229adf" (UID: "e4ef186c-6f92-4602-90f7-e1e472229adf"). InnerVolumeSpecName "kube-api-access-zqxc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:31.770572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.770451    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxc4\" (UniqueName: \"kubernetes.io/projected/e4ef186c-6f92-4602-90f7-e1e472229adf-kube-api-access-zqxc4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:31.770572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:31.770528    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ef186c-6f92-4602-90f7-e1e472229adf-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:32.509970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:32.509944    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"
Nov 01 07:00:32.510163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:32.509969    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r" event={"ID":"e4ef186c-6f92-4602-90f7-e1e472229adf","Type":"ContainerDied","Data":"f6e3c3507c6e4e66f9108944d1fc8b4ae19de49ef6891b26f62bfc9af6b461c7"}
Nov 01 07:00:32.510163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:32.510014    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6e3c3507c6e4e66f9108944d1fc8b4ae19de49ef6891b26f62bfc9af6b461c7"
Nov 01 07:00:32.947046 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:32.947011    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:00:37.643935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:37.643903    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:37.655732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:37.655709    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:38.564011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:38.563985    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:00:39.492272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.492236    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"]
Nov 01 07:00:39.492842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.492827    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e4ef186c-6f92-4602-90f7-e1e472229adf" containerName="revoke-keys"
Nov 01 07:00:39.492886 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.492843    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ef186c-6f92-4602-90f7-e1e472229adf" containerName="revoke-keys"
Nov 01 07:00:39.492947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.492938    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ef186c-6f92-4602-90f7-e1e472229adf" containerName="revoke-keys"
Nov 01 07:00:39.497523 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.497508    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.553305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.553279    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"]
Nov 01 07:00:39.565159 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.565131    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.565303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.565269    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.565447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.565430    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz85f\" (UniqueName: \"kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.598790 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.598759    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:00:39.598790 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.598762    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:00:39.598790 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.598781    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:00:39.599260 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.599246    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:00:39.667113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.667079    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz85f\" (UniqueName: \"kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.667423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.667393    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.667610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.667593    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.668050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.668027    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.669843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.669826    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.723314 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.723281    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz85f\" (UniqueName: \"kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f\") pod \"maas-api-revoke-sub-76c75cb7-mpqhs\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:39.810357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:39.810320    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:40.191460 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:40.191429    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"]
Nov 01 07:00:40.192528 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:00:40.192498    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9ed568_9493_404b_9a34_c9686e3e5a01.slice/crio-07da54e5f947a1166135d7b1085e63cfd677ac837817a8a68951ac658e9f1a08 WatchSource:0}: Error finding container 07da54e5f947a1166135d7b1085e63cfd677ac837817a8a68951ac658e9f1a08: Status 404 returned error can't find the container with id 07da54e5f947a1166135d7b1085e63cfd677ac837817a8a68951ac658e9f1a08
Nov 01 07:00:40.571459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:40.571431    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs" event={"ID":"fe9ed568-9493-404b-9a34-c9686e3e5a01","Type":"ContainerStarted","Data":"34544e5883394cfe271f21b5b83097776d5e9c8cc462a01269add60bef4b1153"}
Nov 01 07:00:40.571843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:40.571494    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs" event={"ID":"fe9ed568-9493-404b-9a34-c9686e3e5a01","Type":"ContainerStarted","Data":"07da54e5f947a1166135d7b1085e63cfd677ac837817a8a68951ac658e9f1a08"}
Nov 01 07:00:40.683735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:40.683690    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs" podStartSLOduration=1.683677582 podStartE2EDuration="1.683677582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:40.681697045 +0000 UTC m=+3052.249503901" watchObservedRunningTime="2026-11-01 07:00:40.683677582 +0000 UTC m=+3052.251484442"
Nov 01 07:00:41.585538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:41.585505    4468 generic.go:356] "Generic (PLEG): container finished" podID="fe9ed568-9493-404b-9a34-c9686e3e5a01" containerID="34544e5883394cfe271f21b5b83097776d5e9c8cc462a01269add60bef4b1153" exitCode=0
Nov 01 07:00:41.585998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:41.585589    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs" event={"ID":"fe9ed568-9493-404b-9a34-c9686e3e5a01","Type":"ContainerDied","Data":"34544e5883394cfe271f21b5b83097776d5e9c8cc462a01269add60bef4b1153"}
Nov 01 07:00:42.750210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.750187    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:42.911530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.911491    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f\" (UniqueName: \"kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f\") pod \"fe9ed568-9493-404b-9a34-c9686e3e5a01\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") "
Nov 01 07:00:42.911725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.911617    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca\") pod \"fe9ed568-9493-404b-9a34-c9686e3e5a01\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") "
Nov 01 07:00:42.911725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.911664    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca\") pod \"fe9ed568-9493-404b-9a34-c9686e3e5a01\" (UID: \"fe9ed568-9493-404b-9a34-c9686e3e5a01\") "
Nov 01 07:00:42.912089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.912059    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca" pod "fe9ed568-9493-404b-9a34-c9686e3e5a01" (UID: "fe9ed568-9493-404b-9a34-c9686e3e5a01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:42.913837 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.913817    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f" pod "fe9ed568-9493-404b-9a34-c9686e3e5a01" (UID: "fe9ed568-9493-404b-9a34-c9686e3e5a01"). InnerVolumeSpecName "kube-api-access-nz85f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:42.913940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:42.913876    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca" pod "fe9ed568-9493-404b-9a34-c9686e3e5a01" (UID: "fe9ed568-9493-404b-9a34-c9686e3e5a01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:00:43.013889 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:43.013827    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz85f\" (UniqueName: \"kubernetes.io/projected/fe9ed568-9493-404b-9a34-c9686e3e5a01-kube-api-access-nz85f\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:43.013889 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:43.013853    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:43.013889 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:43.013863    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9ed568-9493-404b-9a34-c9686e3e5a01-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:43.603752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:43.603705    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs" event={"ID":"fe9ed568-9493-404b-9a34-c9686e3e5a01","Type":"ContainerDied","Data":"07da54e5f947a1166135d7b1085e63cfd677ac837817a8a68951ac658e9f1a08"}
Nov 01 07:00:43.603937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:43.603769    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07da54e5f947a1166135d7b1085e63cfd677ac837817a8a68951ac658e9f1a08"
Nov 01 07:00:43.603937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:43.603817    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"
Nov 01 07:00:44.699694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:44.699661    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"]
Nov 01 07:00:45.401244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.401210    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"]
Nov 01 07:00:45.401540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.401477    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" containerID="cri-o://c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224" gracePeriod=30
Nov 01 07:00:45.405121 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.405098    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"]
Nov 01 07:00:45.405276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.405258    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy" containerID="cri-o://841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7" gracePeriod=30
Nov 01 07:00:45.451350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.451325    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"]
Nov 01 07:00:45.451538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.451519    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy" containerID="cri-o://a1051f1c1bf399d18add31ba8fb58be7b579d53c00c9998c2b9ad98a2ddcee92" gracePeriod=30
Nov 01 07:00:45.454457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.454437    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"]
Nov 01 07:00:45.454663 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:45.454640    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" containerID="cri-o://6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891" gracePeriod=30
Nov 01 07:00:48.794839 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:48.794808    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7jm7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:00:48.795250 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:48.794872    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:00:48.925332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:48.925298    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-dwr2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:00:48.925541 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:48.925360    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:00:50.691738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.691661    4468 generic.go:356] "Generic (PLEG): container finished" podID="769a8b93-1638-4313-887b-128fc0512de5" containerID="c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224" exitCode=0
Nov 01 07:00:50.692198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.691740    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" event={"ID":"769a8b93-1638-4313-887b-128fc0512de5","Type":"ContainerDied","Data":"c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224"}
Nov 01 07:00:50.694057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.693959    4468 generic.go:356] "Generic (PLEG): container finished" podID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerID="841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7" exitCode=0
Nov 01 07:00:50.694057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.694019    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" event={"ID":"f930c3d7-d12f-4acc-8ea8-aa808f252871","Type":"ContainerDied","Data":"841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7"}
Nov 01 07:00:50.695973 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.695949    4468 generic.go:356] "Generic (PLEG): container finished" podID="a4b88da6-fb37-4599-8683-90965863b521" containerID="6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891" exitCode=0
Nov 01 07:00:50.696153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.696105    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" event={"ID":"a4b88da6-fb37-4599-8683-90965863b521","Type":"ContainerDied","Data":"6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891"}
Nov 01 07:00:50.698009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.697931    4468 generic.go:356] "Generic (PLEG): container finished" podID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerID="a1051f1c1bf399d18add31ba8fb58be7b579d53c00c9998c2b9ad98a2ddcee92" exitCode=0
Nov 01 07:00:50.698009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.697977    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" event={"ID":"ccd4b0b1-ce47-419d-a68f-1dc75422da8f","Type":"ContainerDied","Data":"a1051f1c1bf399d18add31ba8fb58be7b579d53c00c9998c2b9ad98a2ddcee92"}
Nov 01 07:00:50.812705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.812676    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 07:00:50.833553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.833353    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 07:00:50.850322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.850291    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 07:00:50.855461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.855436    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 07:00:50.935125 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935088    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.935318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935156    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935186    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935283    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935317    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.935578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935367    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.935578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935382    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.935578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935415    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.935578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935485    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.935578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935522    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.935578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935570    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935595    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935644    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935675    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935707    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935730    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935759    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935794    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935832    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935868    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.935900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935897    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935926    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935958    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936003    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936029    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert\") pod \"769a8b93-1638-4313-887b-128fc0512de5\" (UID: \"769a8b93-1638-4313-887b-128fc0512de5\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936063    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket\") pod \"f930c3d7-d12f-4acc-8ea8-aa808f252871\" (UID: \"f930c3d7-d12f-4acc-8ea8-aa808f252871\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936087    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936116    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token\") pod \"a4b88da6-fb37-4599-8683-90965863b521\" (UID: \"a4b88da6-fb37-4599-8683-90965863b521\") "
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935592    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.935816    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936112    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936348    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.936442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936356    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.937147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936525    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.937147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936696    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.937147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936905    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.937147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.936986    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:50.937147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.937130    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938396    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b88da6-fb37-4599-8683-90965863b521-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938421    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938418    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938436    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938451    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938493    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938507    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938521    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938541    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938556    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938569    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930c3d7-d12f-4acc-8ea8-aa808f252871-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938585    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:50.938822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938637    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.939729 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.938986    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.939729 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.939228    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:50.939729 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.939362    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.939729 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.939454    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:00:50.939931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.939851    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:50.941632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941440    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "kube-api-access-r7dhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:50.941632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941562    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.941632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941564    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "kube-api-access-gvgr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:50.941632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941609    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:00:50.941632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941604    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:00:50.941934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941664    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy" pod "769a8b93-1638-4313-887b-128fc0512de5" (UID: "769a8b93-1638-4313-887b-128fc0512de5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.941934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.941799    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:50.942091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.942073    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy" pod "a4b88da6-fb37-4599-8683-90965863b521" (UID: "a4b88da6-fb37-4599-8683-90965863b521"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:50.942853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:50.942832    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2" pod "f930c3d7-d12f-4acc-8ea8-aa808f252871" (UID: "f930c3d7-d12f-4acc-8ea8-aa808f252871"). InnerVolumeSpecName "kube-api-access-gzlq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:51.039905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.039873    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.039980    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040018    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040089    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040138    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040144    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:51.040361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040220    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040267    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040312    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040343    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5\") pod \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\" (UID: \"ccd4b0b1-ce47-419d-a68f-1dc75422da8f\") "
Nov 01 07:00:51.040604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040428    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:51.040604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040582    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:51.040718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040624    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:51.040718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.040646    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041259    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b88da6-fb37-4599-8683-90965863b521-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041291    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041306    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041319    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041332    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041345    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041358    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzlq2\" (UniqueName: \"kubernetes.io/projected/f930c3d7-d12f-4acc-8ea8-aa808f252871-kube-api-access-gzlq2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041389    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvgr9\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-kube-api-access-gvgr9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041404    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041417    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a8b93-1638-4313-887b-128fc0512de5-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041433    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a8b93-1638-4313-887b-128fc0512de5-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041446    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dhf\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-kube-api-access-r7dhf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041459    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a8b93-1638-4313-887b-128fc0512de5-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041490    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041529    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041544    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b88da6-fb37-4599-8683-90965863b521-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041559    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b88da6-fb37-4599-8683-90965863b521-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041573    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041587    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930c3d7-d12f-4acc-8ea8-aa808f252871-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041600    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a8b93-1638-4313-887b-128fc0512de5-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.041836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.041614    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.042645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.042318    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:00:51.042645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.042433    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:51.042645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.042447    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:00:51.042997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.042980    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5" pod "ccd4b0b1-ce47-419d-a68f-1dc75422da8f" (UID: "ccd4b0b1-ce47-419d-a68f-1dc75422da8f"). InnerVolumeSpecName "kube-api-access-mbrl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:00:51.143892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.143864    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbrl5\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-kube-api-access-mbrl5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.143892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.143892    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.144078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.143903    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.144078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.143912    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd4b0b1-ce47-419d-a68f-1dc75422da8f-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:00:51.705850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.705755    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"
Nov 01 07:00:51.705850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.705773    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6" event={"ID":"ccd4b0b1-ce47-419d-a68f-1dc75422da8f","Type":"ContainerDied","Data":"1c30b3ff7bdceb6810994ddf0633d42c91d90e3fa2a2c82dfe77da95fb68c336"}
Nov 01 07:00:51.705850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.705839    4468 scope.go:122] "RemoveContainer" containerID="a1051f1c1bf399d18add31ba8fb58be7b579d53c00c9998c2b9ad98a2ddcee92"
Nov 01 07:00:51.707370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.707344    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"
Nov 01 07:00:51.707504 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.707380    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x" event={"ID":"769a8b93-1638-4313-887b-128fc0512de5","Type":"ContainerDied","Data":"5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b"}
Nov 01 07:00:51.708968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.708940    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"
Nov 01 07:00:51.708968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.708950    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829" event={"ID":"f930c3d7-d12f-4acc-8ea8-aa808f252871","Type":"ContainerDied","Data":"ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b"}
Nov 01 07:00:51.710572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.710547    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h" event={"ID":"a4b88da6-fb37-4599-8683-90965863b521","Type":"ContainerDied","Data":"f1f44025f753124d1a47e54cbeceff5622318c1c33fc3131e5e711d7ab8142d6"}
Nov 01 07:00:51.710670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.710556    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"
Nov 01 07:00:51.717185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.717141    4468 scope.go:122] "RemoveContainer" containerID="c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224"
Nov 01 07:00:51.722958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.722940    4468 scope.go:122] "RemoveContainer" containerID="c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224"
Nov 01 07:00:51.729144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.728451    4468 scope.go:122] "RemoveContainer" containerID="841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7"
Nov 01 07:00:51.729144 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:51.728905    4468 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-7d9897b54c-7jm7x_openshift-ingress_769a8b93-1638-4313-887b-128fc0512de5_0 in pod sandbox 5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b from index: no such id: 'c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224'" containerID="c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224"
Nov 01 07:00:51.729144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.728937    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-7d9897b54c-7jm7x_openshift-ingress_769a8b93-1638-4313-887b-128fc0512de5_0 in pod sandbox 5610984648e70aa783eca30087ed32307569b2971731f72babc4345aeef7f97b from index: no such id: 'c264f1cbb8da03e59adc560677abab849a6ae47fb6edb82c22c3d55a55a31224'"
Nov 01 07:00:51.729144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.728953    4468 scope.go:122] "RemoveContainer" containerID="841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7"
Nov 01 07:00:51.736749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.736727    4468 scope.go:122] "RemoveContainer" containerID="6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891"
Nov 01 07:00:51.737412 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:51.737387    4468 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-7d9897b54c-g2829_openshift-ingress_f930c3d7-d12f-4acc-8ea8-aa808f252871_0 in pod sandbox ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b from index: no such id: '841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7'" containerID="841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7"
Nov 01 07:00:51.737551 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:51.737414    4468 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-7d9897b54c-g2829_openshift-ingress_f930c3d7-d12f-4acc-8ea8-aa808f252871_0 in pod sandbox ee68a33cbb66bfe4a7709d52633cfeb194e32306975b28db9a66b2b27423887b from index: no such id: '841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7'" containerID="841ad15d58e9de2830f7ee5a10fb910dd3358919e1a2ecd48ed1a84b763888d7"
Nov 01 07:00:51.737551 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.737435    4468 scope.go:122] "RemoveContainer" containerID="8627f9bf66c3037926f18dada734f80768197fb04bf01889b7200d9dfc2277af"
Nov 01 07:00:51.749667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.749649    4468 scope.go:122] "RemoveContainer" containerID="62d6b8057462d932f2f4f81827477f6400252fc3900def4215a0fbcb6af2bcb1"
Nov 01 07:00:51.757885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.757869    4468 scope.go:122] "RemoveContainer" containerID="6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891"
Nov 01 07:00:51.758123 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:51.758108    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891\": container with ID starting with 6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891 not found: ID does not exist" containerID="6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891"
Nov 01 07:00:51.758174 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:00:51.758128    4468 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891\": rpc error: code = NotFound desc = could not find container \"6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891\": container with ID starting with 6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891 not found: ID does not exist" containerID="6cb528f8d1fe968fdd96ad5565010732cd6331d95fd38272041daba3d2e5f891"
Nov 01 07:00:51.869963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.869936    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"]
Nov 01 07:00:51.921030 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:51.920996    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-dwr2h"]
Nov 01 07:00:52.028372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.028291    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"]
Nov 01 07:00:52.034090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.034066    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7jm7x"]
Nov 01 07:00:52.143891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.143862    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"]
Nov 01 07:00:52.150150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.150120    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-p4lr6"]
Nov 01 07:00:52.259855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.259610    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"]
Nov 01 07:00:52.264314 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.264289    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-g2829"]
Nov 01 07:00:52.590773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.590738    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769a8b93-1638-4313-887b-128fc0512de5" path="/var/lib/kubelet/pods/769a8b93-1638-4313-887b-128fc0512de5/volumes"
Nov 01 07:00:52.591244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.591230    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b88da6-fb37-4599-8683-90965863b521" path="/var/lib/kubelet/pods/a4b88da6-fb37-4599-8683-90965863b521/volumes"
Nov 01 07:00:52.591703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.591689    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" path="/var/lib/kubelet/pods/ccd4b0b1-ce47-419d-a68f-1dc75422da8f/volumes"
Nov 01 07:00:52.592439 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:52.592425    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" path="/var/lib/kubelet/pods/f930c3d7-d12f-4acc-8ea8-aa808f252871/volumes"
Nov 01 07:00:59.720220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:59.720152    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl" podUID="d26d1d65-0b4a-4484-bcef-32c77f491dd5" containerName="main" containerID="cri-o://b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604" gracePeriod=45
Nov 01 07:00:59.979070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:00:59.979014    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:01:00.053583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053558    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home\") pod \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") "
Nov 01 07:01:00.053741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053606    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc\" (UniqueName: \"kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc\") pod \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") "
Nov 01 07:01:00.053741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053694    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm\") pod \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") "
Nov 01 07:01:00.053820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053783    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs\") pod \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") "
Nov 01 07:01:00.053820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053794    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home" pod "d26d1d65-0b4a-4484-bcef-32c77f491dd5" (UID: "d26d1d65-0b4a-4484-bcef-32c77f491dd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:00.053888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053852    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir\") pod \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") "
Nov 01 07:01:00.053960 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.053943    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache\") pod \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\" (UID: \"d26d1d65-0b4a-4484-bcef-32c77f491dd5\") "
Nov 01 07:01:00.054132 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.054113    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir" pod "d26d1d65-0b4a-4484-bcef-32c77f491dd5" (UID: "d26d1d65-0b4a-4484-bcef-32c77f491dd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:00.054223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.054156    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache" pod "d26d1d65-0b4a-4484-bcef-32c77f491dd5" (UID: "d26d1d65-0b4a-4484-bcef-32c77f491dd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:00.054438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.054422    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:00.054519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.054440    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:00.054519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.054449    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:00.056067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.056048    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs" pod "d26d1d65-0b4a-4484-bcef-32c77f491dd5" (UID: "d26d1d65-0b4a-4484-bcef-32c77f491dd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:00.056461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.056437    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm" pod "d26d1d65-0b4a-4484-bcef-32c77f491dd5" (UID: "d26d1d65-0b4a-4484-bcef-32c77f491dd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:00.056584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.056493    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc" pod "d26d1d65-0b4a-4484-bcef-32c77f491dd5" (UID: "d26d1d65-0b4a-4484-bcef-32c77f491dd5"). InnerVolumeSpecName "kube-api-access-ks8wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:00.156254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.156222    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26d1d65-0b4a-4484-bcef-32c77f491dd5-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:00.156254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.156250    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26d1d65-0b4a-4484-bcef-32c77f491dd5-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:00.156254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.156260    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8wc\" (UniqueName: \"kubernetes.io/projected/d26d1d65-0b4a-4484-bcef-32c77f491dd5-kube-api-access-ks8wc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:00.766413 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.766377    4468 generic.go:356] "Generic (PLEG): container finished" podID="d26d1d65-0b4a-4484-bcef-32c77f491dd5" containerID="b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604" exitCode=0
Nov 01 07:01:00.766856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.766430    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"
Nov 01 07:01:00.766856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.766461    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl" event={"ID":"d26d1d65-0b4a-4484-bcef-32c77f491dd5","Type":"ContainerDied","Data":"b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604"}
Nov 01 07:01:00.766856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.766532    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl" event={"ID":"d26d1d65-0b4a-4484-bcef-32c77f491dd5","Type":"ContainerDied","Data":"a148353f161d41196d921a4b6aed14d9c4079a8ec8e9c1e413113adadddecbfc"}
Nov 01 07:01:00.766856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.766558    4468 scope.go:122] "RemoveContainer" containerID="b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604"
Nov 01 07:01:00.775895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.775874    4468 scope.go:122] "RemoveContainer" containerID="b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604"
Nov 01 07:01:00.776178 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:00.776154    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604\": container with ID starting with b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604 not found: ID does not exist" containerID="b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604"
Nov 01 07:01:00.776233 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.776188    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604"} err="failed to get container status \"b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604\": rpc error: code = NotFound desc = could not find container \"b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604\": container with ID starting with b40ad360972beaf7887db99c9e54d12db27eeeb51e38584e25d0823a56fdc604 not found: ID does not exist"
Nov 01 07:01:00.930030 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.929943    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"]
Nov 01 07:01:00.979041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:00.979008    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0cb4ef/explicit-model-0cb4ef-kserve-7c546f5479-54btl"]
Nov 01 07:01:02.590595 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:02.590528    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26d1d65-0b4a-4484-bcef-32c77f491dd5" path="/var/lib/kubelet/pods/d26d1d65-0b4a-4484-bcef-32c77f491dd5/volumes"
Nov 01 07:01:02.948712 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:02.948622    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:01:09.044713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:09.044677    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"]
Nov 01 07:01:09.093965 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:09.093926    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-665e906c-bhkns"]
Nov 01 07:01:10.590652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:10.590611    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b149084-f7e3-4d91-8b29-945ecc8dbb68" path="/var/lib/kubelet/pods/3b149084-f7e3-4d91-8b29-945ecc8dbb68/volumes"
Nov 01 07:01:16.947776 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:16.947730    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:01:27.967483 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:27.967382    4468 generic.go:356] "Generic (PLEG): container finished" podID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerID="8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865" exitCode=0
Nov 01 07:01:27.967483 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:27.967454    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" event={"ID":"b24aeff4-8c1e-4638-ae19-8c09010af0f6","Type":"ContainerDied","Data":"8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"}
Nov 01 07:01:27.968014 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:27.967954    4468 scope.go:122] "RemoveContainer" containerID="8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"
Nov 01 07:01:27.970149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:27.970107    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:01:27.970277 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:27.970207    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:01:27.970327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:27.970273    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:01:28.074824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.074790    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-c60e8b"]
Nov 01 07:01:28.075342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075326    4468 cpu_manager.go:414] "containerMap: removing container" podUID="fe9ed568-9493-404b-9a34-c9686e3e5a01" containerName="revoke-keys"
Nov 01 07:01:28.075342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075340    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9ed568-9493-404b-9a34-c9686e3e5a01" containerName="revoke-keys"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075373    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075378    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075389    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075394    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075404    4468 cpu_manager.go:414] "containerMap: removing container" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075409    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075416    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d26d1d65-0b4a-4484-bcef-32c77f491dd5" containerName="main"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075420    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26d1d65-0b4a-4484-bcef-32c77f491dd5" containerName="main"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075428    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075432    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy"
Nov 01 07:01:28.075553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075555    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd4b0b1-ce47-419d-a68f-1dc75422da8f" containerName="istio-proxy"
Nov 01 07:01:28.076169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075568    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f930c3d7-d12f-4acc-8ea8-aa808f252871" containerName="istio-proxy"
Nov 01 07:01:28.076169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075576    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="769a8b93-1638-4313-887b-128fc0512de5" containerName="istio-proxy"
Nov 01 07:01:28.076169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075582    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26d1d65-0b4a-4484-bcef-32c77f491dd5" containerName="main"
Nov 01 07:01:28.076169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075588    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b88da6-fb37-4599-8683-90965863b521" containerName="istio-proxy"
Nov 01 07:01:28.076169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.075598    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9ed568-9493-404b-9a34-c9686e3e5a01" containerName="revoke-keys"
Nov 01 07:01:28.078810 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.078784    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:28.131538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.131097    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-c60e8b"]
Nov 01 07:01:28.131538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.131215    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\""
Nov 01 07:01:28.154498 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.153216    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z98g9\" (UniqueName: \"kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9\") pod \"test-curl-7695-c60e8b\" (UID: \"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8\") " pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:28.259663 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.259614    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z98g9\" (UniqueName: \"kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9\") pod \"test-curl-7695-c60e8b\" (UID: \"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8\") " pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:28.315881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.315839    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z98g9\" (UniqueName: \"kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9\") pod \"test-curl-7695-c60e8b\" (UID: \"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8\") " pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:28.392563 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.392521    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:28.774177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.774148    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-c60e8b"]
Nov 01 07:01:28.775670 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:01:28.775641    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f8c9fe3_f383_44b6_9e00_3e46f55f3ed8.slice/crio-6c77b8d42af17ae6a7a637c8d8f7b379b5fc39b479eaf313d5f86dcef4c589c2 WatchSource:0}: Error finding container 6c77b8d42af17ae6a7a637c8d8f7b379b5fc39b479eaf313d5f86dcef4c589c2: Status 404 returned error can't find the container with id 6c77b8d42af17ae6a7a637c8d8f7b379b5fc39b479eaf313d5f86dcef4c589c2
Nov 01 07:01:28.979634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.979595    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" event={"ID":"b24aeff4-8c1e-4638-ae19-8c09010af0f6","Type":"ContainerStarted","Data":"835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3"}
Nov 01 07:01:28.983152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.983115    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-c60e8b" event={"ID":"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8","Type":"ContainerStarted","Data":"22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5"}
Nov 01 07:01:28.983419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:28.983390    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-c60e8b" event={"ID":"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8","Type":"ContainerStarted","Data":"6c77b8d42af17ae6a7a637c8d8f7b379b5fc39b479eaf313d5f86dcef4c589c2"}
Nov 01 07:01:29.217493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:29.211054    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7695-c60e8b" podStartSLOduration=1.211031925 podStartE2EDuration="1.211031925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:01:29.208505399 +0000 UTC m=+3100.776312230" watchObservedRunningTime="2026-11-01 07:01:29.211031925 +0000 UTC m=+3100.778838756"
Nov 01 07:01:29.968764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:29.968715    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 07:01:29.970108 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:29.970080    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 01 07:01:29.970231 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:29.970133    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 01 07:01:30.663286 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:30.663246    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-c60e8b"]
Nov 01 07:01:30.668816 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:30.668781    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-c60e8b"]
Nov 01 07:01:30.968393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:30.968312    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 01 07:01:30.968393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:30.968368    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 01 07:01:30.996540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:30.996499    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7695-c60e8b" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" containerName="test-curl-7695-c60e8b" containerID="cri-o://22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5" gracePeriod=2
Nov 01 07:01:31.052536 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:31.052458    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-c60e8b\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:31.531665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.531624    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"]
Nov 01 07:01:31.532604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.532577    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" containerName="test-curl-7695-c60e8b"
Nov 01 07:01:31.532604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.532603    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" containerName="test-curl-7695-c60e8b"
Nov 01 07:01:31.532824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.532758    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" containerName="test-curl-7695-c60e8b"
Nov 01 07:01:31.535984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.535963    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.587068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.587031    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"]
Nov 01 07:01:31.633418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.633274    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:01:31.633418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.633285    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:01:31.633418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.633311    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:01:31.637175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.637147    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:01:31.654621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.654579    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.654849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.654735    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khtq\" (UniqueName: \"kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.654849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.654773    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.683810 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.683779    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 07:01:31.743411 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:31.743360    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-c60e8b\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:31.756096 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.756059    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khtq\" (UniqueName: \"kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.756279 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.756110    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.756324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.756288    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.757057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.756960    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.758823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.758798    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.833419    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khtq\" (UniqueName: \"kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq\") pod \"maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.852659 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.852618    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:31.968695 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.968661    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 01 07:01:31.968880 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:31.968714    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 01 07:01:32.254536 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:32.254508    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"]
Nov 01 07:01:32.256397 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:01:32.256359    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2a7055_d2b5_409f_aec3_6a4a9a30b17e.slice/crio-e5f45f82716095e8b26d912c1415ecc5852ab7e9c72b2348930cfab5f7937000 WatchSource:0}: Error finding container e5f45f82716095e8b26d912c1415ecc5852ab7e9c72b2348930cfab5f7937000: Status 404 returned error can't find the container with id e5f45f82716095e8b26d912c1415ecc5852ab7e9c72b2348930cfab5f7937000
Nov 01 07:01:32.957882 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:32.957843    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:01:32.968348 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:32.968315    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 01 07:01:32.968559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:32.968368    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 01 07:01:33.035803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.035685    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4" event={"ID":"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e","Type":"ContainerStarted","Data":"a70c82675d783e6bea5fb2f3dc1251fc8f46ff9fb2f62c1d03bc42dbd27a69d4"}
Nov 01 07:01:33.035803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.035753    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4" event={"ID":"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e","Type":"ContainerStarted","Data":"e5f45f82716095e8b26d912c1415ecc5852ab7e9c72b2348930cfab5f7937000"}
Nov 01 07:01:33.150571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.150496    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4" podStartSLOduration=2.150458498 podStartE2EDuration="2.150458498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:01:33.146904505 +0000 UTC m=+3104.714711334" watchObservedRunningTime="2026-11-01 07:01:33.150458498 +0000 UTC m=+3104.718265335"
Nov 01 07:01:33.259199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.259129    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:33.377895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.377867    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9\" (UniqueName: \"kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9\") pod \"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8\" (UID: \"3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8\") "
Nov 01 07:01:33.380091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.380055    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9" pod "3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" (UID: "3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8"). InnerVolumeSpecName "kube-api-access-z98g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:33.480116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.480066    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z98g9\" (UniqueName: \"kubernetes.io/projected/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8-kube-api-access-z98g9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:33.968549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.968508    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 01 07:01:33.969068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:33.968581    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 01 07:01:34.043711 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.043617    4468 generic.go:356] "Generic (PLEG): container finished" podID="6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" containerID="a70c82675d783e6bea5fb2f3dc1251fc8f46ff9fb2f62c1d03bc42dbd27a69d4" exitCode=0
Nov 01 07:01:34.043879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.043705    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4" event={"ID":"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e","Type":"ContainerDied","Data":"a70c82675d783e6bea5fb2f3dc1251fc8f46ff9fb2f62c1d03bc42dbd27a69d4"}
Nov 01 07:01:34.045109 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.045080    4468 generic.go:356] "Generic (PLEG): container finished" podID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" containerID="22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5" exitCode=137
Nov 01 07:01:34.045267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.045134    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:34.045267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.045143    4468 scope.go:122] "RemoveContainer" containerID="22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5"
Nov 01 07:01:34.055645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.055620    4468 scope.go:122] "RemoveContainer" containerID="22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5"
Nov 01 07:01:34.055997 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:34.055978    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5\": container with ID starting with 22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5 not found: ID does not exist" containerID="22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5"
Nov 01 07:01:34.056046 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.056006    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5"} err="failed to get container status \"22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5\": rpc error: code = NotFound desc = could not find container \"22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5\": container with ID starting with 22e7f7c6cf5d26db175f5a5b88ad0dc43f09df22f6fdbeb31847a5c789d5a7e5 not found: ID does not exist"
Nov 01 07:01:34.209700 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:34.208487    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-c60e8b\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" pod="opendatahub/test-curl-7695-c60e8b"
Nov 01 07:01:34.591667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.591632    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8" path="/var/lib/kubelet/pods/3f8c9fe3-f383-44b6-9e00-3e46f55f3ed8/volumes"
Nov 01 07:01:34.973721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.973635    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 07:01:34.974817 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:34.974793    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 07:01:35.220505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.220438    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:35.305574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.305534    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq\" (UniqueName: \"kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq\") pod \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") "
Nov 01 07:01:35.305785 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.305594    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca\") pod \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") "
Nov 01 07:01:35.305785 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.305742    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca\") pod \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\" (UID: \"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e\") "
Nov 01 07:01:35.306008 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.305985    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca" pod "6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" (UID: "6b2a7055-d2b5-409f-aec3-6a4a9a30b17e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:35.306350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.306330    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:35.308120 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.308089    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca" pod "6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" (UID: "6b2a7055-d2b5-409f-aec3-6a4a9a30b17e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:01:35.308627 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.308597    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq" pod "6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" (UID: "6b2a7055-d2b5-409f-aec3-6a4a9a30b17e"). InnerVolumeSpecName "kube-api-access-9khtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:35.407635 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.407598    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:35.407635 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:35.407636    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9khtq\" (UniqueName: \"kubernetes.io/projected/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e-kube-api-access-9khtq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:36.061352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:36.061320    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"
Nov 01 07:01:36.061897 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:36.061349    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4" event={"ID":"6b2a7055-d2b5-409f-aec3-6a4a9a30b17e","Type":"ContainerDied","Data":"e5f45f82716095e8b26d912c1415ecc5852ab7e9c72b2348930cfab5f7937000"}
Nov 01 07:01:36.061897 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:36.061403    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f45f82716095e8b26d912c1415ecc5852ab7e9c72b2348930cfab5f7937000"
Nov 01 07:01:40.928323 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:40.928284    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"]
Nov 01 07:01:40.928865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:40.928581    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy" containerID="cri-o://24a52f290364c95c098e732928a178002b994216538f62d8ca2f811920bafe6f" gracePeriod=30
Nov 01 07:01:46.159447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.159408    4468 generic.go:356] "Generic (PLEG): container finished" podID="838d3f5d-009d-4679-8a14-3347e50680e0" containerID="24a52f290364c95c098e732928a178002b994216538f62d8ca2f811920bafe6f" exitCode=0
Nov 01 07:01:46.159885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.159451    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" event={"ID":"838d3f5d-009d-4679-8a14-3347e50680e0","Type":"ContainerDied","Data":"24a52f290364c95c098e732928a178002b994216538f62d8ca2f811920bafe6f"}
Nov 01 07:01:46.210669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.210596    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 07:01:46.256272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256228    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256350    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256403    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256441    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256556    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256624    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256710    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256748    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.256925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256807    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket\") pod \"838d3f5d-009d-4679-8a14-3347e50680e0\" (UID: \"838d3f5d-009d-4679-8a14-3347e50680e0\") "
Nov 01 07:01:46.257039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256870    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:46.257039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.256967    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:46.257223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257200    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:46.257389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257364    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:46.257625 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257594    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:46.257738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257687    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.257738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257710    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.257738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257726    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838d3f5d-009d-4679-8a14-3347e50680e0-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.257932 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.257741    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.258937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.258910    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:01:46.259024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.258985    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:46.259433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.259411    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:46.259433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.259422    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f" pod "838d3f5d-009d-4679-8a14-3347e50680e0" (UID: "838d3f5d-009d-4679-8a14-3347e50680e0"). InnerVolumeSpecName "kube-api-access-rgf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:46.358983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.358941    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.358983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.358977    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.358983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.358991    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838d3f5d-009d-4679-8a14-3347e50680e0-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.359246 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.359003    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838d3f5d-009d-4679-8a14-3347e50680e0-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:46.359246 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:46.359015    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgf4f\" (UniqueName: \"kubernetes.io/projected/838d3f5d-009d-4679-8a14-3347e50680e0-kube-api-access-rgf4f\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:47.168198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:47.168168    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"
Nov 01 07:01:47.168198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:47.168188    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng" event={"ID":"838d3f5d-009d-4679-8a14-3347e50680e0","Type":"ContainerDied","Data":"0f722946148cedc5d9496ad983c02a2914fcd584add77e66fbb9fde79486f96e"}
Nov 01 07:01:47.168829 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:47.168260    4468 scope.go:122] "RemoveContainer" containerID="24a52f290364c95c098e732928a178002b994216538f62d8ca2f811920bafe6f"
Nov 01 07:01:47.386790 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:47.386747    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"]
Nov 01 07:01:47.436315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:47.436210    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-cc8ng"]
Nov 01 07:01:48.591598 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:48.591559    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" path="/var/lib/kubelet/pods/838d3f5d-009d-4679-8a14-3347e50680e0/volumes"
Nov 01 07:01:51.482565 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:51.482522    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"]
Nov 01 07:01:51.483096 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:51.482747    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" containerID="cri-o://4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812" gracePeriod=30
Nov 01 07:01:55.101817 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:55.101777    4468 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:01:55.102247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:55.101849    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:01:56.756641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.756579    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 07:01:56.895543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895487    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.895762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895557    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.895762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895626    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.895762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895712    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.895959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895831    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.895959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895892    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.895959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895890    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:56.895959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895916    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:56.895959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895952    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.896222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.895986    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.896222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896040    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert\") pod \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\" (UID: \"0adc3f32-2220-4c25-849e-c873a4ff8c2c\") "
Nov 01 07:01:56.896222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896204    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:56.896380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896230    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:56.896440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896418    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:01:56.896902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896875    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.896902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896902    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.897110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896918    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.897110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896933    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.897110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.896947    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.898307 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.898281    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:56.898418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.898338    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "kube-api-access-7w7h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:01:56.898521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.898457    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:01:56.898521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.898485    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo" pod "0adc3f32-2220-4c25-849e-c873a4ff8c2c" (UID: "0adc3f32-2220-4c25-849e-c873a4ff8c2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:01:56.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.998646    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7h2\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-kube-api-access-7w7h2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.998679    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.998689    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:56.998685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:56.998702    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0adc3f32-2220-4c25-849e-c873a4ff8c2c-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:01:57.250651 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.250610    4468 generic.go:356] "Generic (PLEG): container finished" podID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerID="4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812" exitCode=0
Nov 01 07:01:57.250890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.250674    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"
Nov 01 07:01:57.250890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.250719    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" event={"ID":"0adc3f32-2220-4c25-849e-c873a4ff8c2c","Type":"ContainerDied","Data":"4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812"}
Nov 01 07:01:57.250890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.250781    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7" event={"ID":"0adc3f32-2220-4c25-849e-c873a4ff8c2c","Type":"ContainerDied","Data":"83e1dfea4f7ffaeff7174d07a4d47cd643271f475a8e33d06c8c7b9ee71eeede"}
Nov 01 07:01:57.250890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.250810    4468 scope.go:122] "RemoveContainer" containerID="4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812"
Nov 01 07:01:57.262126 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.262107    4468 scope.go:122] "RemoveContainer" containerID="944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"
Nov 01 07:01:57.272219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.272195    4468 scope.go:122] "RemoveContainer" containerID="4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812"
Nov 01 07:01:57.272623 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:57.272597    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812\": container with ID starting with 4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812 not found: ID does not exist" containerID="4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812"
Nov 01 07:01:57.272738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.272632    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812"} err="failed to get container status \"4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812\": rpc error: code = NotFound desc = could not find container \"4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812\": container with ID starting with 4d25780bd83bc7f8212595d5699946292341190a5552817e45565b544e7b0812 not found: ID does not exist"
Nov 01 07:01:57.272738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.272651    4468 scope.go:122] "RemoveContainer" containerID="944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"
Nov 01 07:01:57.272938 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:01:57.272921    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047\": container with ID starting with 944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047 not found: ID does not exist" containerID="944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"
Nov 01 07:01:57.272981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.272945    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047"} err="failed to get container status \"944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047\": rpc error: code = NotFound desc = could not find container \"944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047\": container with ID starting with 944564530a000bd896a7a79889fc866d1e7f137f1bba90ddad06d9e542ab2047 not found: ID does not exist"
Nov 01 07:01:57.414774 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.414739    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"]
Nov 01 07:01:57.464732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:57.464702    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-5n8j7"]
Nov 01 07:01:58.590745 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:01:58.590709    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" path="/var/lib/kubelet/pods/0adc3f32-2220-4c25-849e-c873a4ff8c2c/volumes"
Nov 01 07:02:02.949003 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:02:02.948962    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:02:15.834931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.834894    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"]
Nov 01 07:02:15.835515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835500    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy"
Nov 01 07:02:15.835571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835517    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy"
Nov 01 07:02:15.835571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835527    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy"
Nov 01 07:02:15.835571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835532    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy"
Nov 01 07:02:15.835571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835555    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" containerName="revoke-keys"
Nov 01 07:02:15.835571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835561    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" containerName="revoke-keys"
Nov 01 07:02:15.835571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835571    4468 cpu_manager.go:414] "containerMap: removing container" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy"
Nov 01 07:02:15.835767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835576    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy"
Nov 01 07:02:15.835767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835659    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" containerName="revoke-keys"
Nov 01 07:02:15.835767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835669    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="838d3f5d-009d-4679-8a14-3347e50680e0" containerName="istio-proxy"
Nov 01 07:02:15.835767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835681    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy"
Nov 01 07:02:15.835767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.835687    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adc3f32-2220-4c25-849e-c873a4ff8c2c" containerName="istio-proxy"
Nov 01 07:02:15.838299 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.838278    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:15.896320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.896286    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"]
Nov 01 07:02:15.926929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.926885    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkp2q\" (UniqueName: \"kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:15.927165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.927147    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:15.927228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.927192    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:15.941883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.941850    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:15.941883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.941870    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:15.941883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.941858    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:15.942209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:15.941944    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:02:16.028703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.028669    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkp2q\" (UniqueName: \"kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.028905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.028854    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.028971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.028908    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.029599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.029572    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.031701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.031678    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.085459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.085376    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkp2q\" (UniqueName: \"kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q\") pod \"maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.152434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.152393    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:16.530522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:16.530491    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"]
Nov 01 07:02:16.532141 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:02:16.532110    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fddf3f4_31ee_4412_88c8_f4109195290c.slice/crio-92832bbbaf9f3e6566ad935305286d6463c589956742b858b33160d09bed7bc8 WatchSource:0}: Error finding container 92832bbbaf9f3e6566ad935305286d6463c589956742b858b33160d09bed7bc8: Status 404 returned error can't find the container with id 92832bbbaf9f3e6566ad935305286d6463c589956742b858b33160d09bed7bc8
Nov 01 07:02:17.385792 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:17.385757    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26" event={"ID":"0fddf3f4-31ee-4412-88c8-f4109195290c","Type":"ContainerStarted","Data":"44f14f24cffd6153f5439ac8eb5324d721087c377cf4ac7af7c9a954832f253f"}
Nov 01 07:02:17.386336 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:17.385815    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26" event={"ID":"0fddf3f4-31ee-4412-88c8-f4109195290c","Type":"ContainerStarted","Data":"92832bbbaf9f3e6566ad935305286d6463c589956742b858b33160d09bed7bc8"}
Nov 01 07:02:17.500250 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:17.500197    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26" podStartSLOduration=2.500176459 podStartE2EDuration="2.500176459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:02:17.496616076 +0000 UTC m=+3149.064422918" watchObservedRunningTime="2026-11-01 07:02:17.500176459 +0000 UTC m=+3149.067983288"
Nov 01 07:02:18.409229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:18.409191    4468 generic.go:356] "Generic (PLEG): container finished" podID="0fddf3f4-31ee-4412-88c8-f4109195290c" containerID="44f14f24cffd6153f5439ac8eb5324d721087c377cf4ac7af7c9a954832f253f" exitCode=0
Nov 01 07:02:18.409922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:18.409416    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26" event={"ID":"0fddf3f4-31ee-4412-88c8-f4109195290c","Type":"ContainerDied","Data":"44f14f24cffd6153f5439ac8eb5324d721087c377cf4ac7af7c9a954832f253f"}
Nov 01 07:02:19.583860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.583833    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:19.695605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.695201    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q\" (UniqueName: \"kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q\") pod \"0fddf3f4-31ee-4412-88c8-f4109195290c\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") "
Nov 01 07:02:19.695605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.695440    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca\") pod \"0fddf3f4-31ee-4412-88c8-f4109195290c\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") "
Nov 01 07:02:19.697189 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.696002    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca\") pod \"0fddf3f4-31ee-4412-88c8-f4109195290c\" (UID: \"0fddf3f4-31ee-4412-88c8-f4109195290c\") "
Nov 01 07:02:19.697189 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.696972    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca" pod "0fddf3f4-31ee-4412-88c8-f4109195290c" (UID: "0fddf3f4-31ee-4412-88c8-f4109195290c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:19.698696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.698601    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca" pod "0fddf3f4-31ee-4412-88c8-f4109195290c" (UID: "0fddf3f4-31ee-4412-88c8-f4109195290c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:19.702522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.702458    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q" pod "0fddf3f4-31ee-4412-88c8-f4109195290c" (UID: "0fddf3f4-31ee-4412-88c8-f4109195290c"). InnerVolumeSpecName "kube-api-access-xkp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:19.798090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.798054    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:19.798090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.798088    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fddf3f4-31ee-4412-88c8-f4109195290c-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:19.798090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:19.798099    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkp2q\" (UniqueName: \"kubernetes.io/projected/0fddf3f4-31ee-4412-88c8-f4109195290c-kube-api-access-xkp2q\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:20.423936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:20.423902    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"
Nov 01 07:02:20.423936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:20.423924    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26" event={"ID":"0fddf3f4-31ee-4412-88c8-f4109195290c","Type":"ContainerDied","Data":"92832bbbaf9f3e6566ad935305286d6463c589956742b858b33160d09bed7bc8"}
Nov 01 07:02:20.424210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:20.423980    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92832bbbaf9f3e6566ad935305286d6463c589956742b858b33160d09bed7bc8"
Nov 01 07:02:27.990769 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:27.990731    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"]
Nov 01 07:02:27.991640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:27.991603    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0fddf3f4-31ee-4412-88c8-f4109195290c" containerName="revoke-keys"
Nov 01 07:02:27.991640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:27.991640    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fddf3f4-31ee-4412-88c8-f4109195290c" containerName="revoke-keys"
Nov 01 07:02:27.991871 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:27.991811    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fddf3f4-31ee-4412-88c8-f4109195290c" containerName="revoke-keys"
Nov 01 07:02:27.994901 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:27.994882    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.049904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.049872    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"]
Nov 01 07:02:28.095332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.095296    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:02:28.095332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.095296    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:02:28.095332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.095315    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:02:28.096355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.096336    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:02:28.111081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.111052    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qjb\" (UniqueName: \"kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.111203 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.111106    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.111284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.111222    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.212786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.212744    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qjb\" (UniqueName: \"kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.212993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.212812    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.212993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.212857    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.213450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.213423    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.215368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.215343    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.270386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.270319    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qjb\" (UniqueName: \"kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb\") pod \"maas-api-revoke-sub-9a2a2a66-zbqg5\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.307256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.307225    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:28.690441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:28.690386    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"]
Nov 01 07:02:28.712096 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:02:28.712061    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7b7ff5_da13_49fb_9e5a_dfa22ffb70ae.slice/crio-b375f0c994952941dfb07a1377f766bfb1878d6490ae43db0f9c685e25d31cf0 WatchSource:0}: Error finding container b375f0c994952941dfb07a1377f766bfb1878d6490ae43db0f9c685e25d31cf0: Status 404 returned error can't find the container with id b375f0c994952941dfb07a1377f766bfb1878d6490ae43db0f9c685e25d31cf0
Nov 01 07:02:29.481249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:29.481212    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5" event={"ID":"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae","Type":"ContainerStarted","Data":"7b7d728ae67d39570aab9a7b052eeb4b812e9c437ef350e1d7f64cddcaad0934"}
Nov 01 07:02:29.481684 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:29.481290    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5" event={"ID":"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae","Type":"ContainerStarted","Data":"b375f0c994952941dfb07a1377f766bfb1878d6490ae43db0f9c685e25d31cf0"}
Nov 01 07:02:29.598247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:29.598180    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5" podStartSLOduration=2.5981664159999998 podStartE2EDuration="2.598166416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:02:29.594493603 +0000 UTC m=+3161.162300434" watchObservedRunningTime="2026-11-01 07:02:29.598166416 +0000 UTC m=+3161.165973267"
Nov 01 07:02:30.489361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:30.489323    4468 generic.go:356] "Generic (PLEG): container finished" podID="2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" containerID="7b7d728ae67d39570aab9a7b052eeb4b812e9c437ef350e1d7f64cddcaad0934" exitCode=0
Nov 01 07:02:30.489821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:30.489367    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5" event={"ID":"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae","Type":"ContainerDied","Data":"7b7d728ae67d39570aab9a7b052eeb4b812e9c437ef350e1d7f64cddcaad0934"}
Nov 01 07:02:31.643548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.643521    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:31.758726 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.758636    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca\") pod \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") "
Nov 01 07:02:31.758906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.758726    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb\" (UniqueName: \"kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb\") pod \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") "
Nov 01 07:02:31.758993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.758933    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca\") pod \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\" (UID: \"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae\") "
Nov 01 07:02:31.759338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.759308    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca" pod "2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" (UID: "2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:31.759560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.759530    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:31.761270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.761218    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb" pod "2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" (UID: "2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae"). InnerVolumeSpecName "kube-api-access-r8qjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:31.761610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.761586    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca" pod "2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" (UID: "2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:02:31.860325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.860291    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8qjb\" (UniqueName: \"kubernetes.io/projected/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-kube-api-access-r8qjb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:31.860325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:31.860321    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:32.503642 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:32.503608    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"
Nov 01 07:02:32.503821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:32.503636    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5" event={"ID":"2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae","Type":"ContainerDied","Data":"b375f0c994952941dfb07a1377f766bfb1878d6490ae43db0f9c685e25d31cf0"}
Nov 01 07:02:32.503821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:32.503684    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b375f0c994952941dfb07a1377f766bfb1878d6490ae43db0f9c685e25d31cf0"
Nov 01 07:02:32.948426 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:02:32.948391    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:02:39.810648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:39.810612    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"]
Nov 01 07:02:39.858203 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:39.858169    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0cb4ef-9zg26"]
Nov 01 07:02:40.030135 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:40.030098    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"]
Nov 01 07:02:40.030410 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:40.030381    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy" containerID="cri-o://bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5" gracePeriod=30
Nov 01 07:02:40.590346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:40.590312    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fddf3f4-31ee-4412-88c8-f4109195290c" path="/var/lib/kubelet/pods/0fddf3f4-31ee-4412-88c8-f4109195290c/volumes"
Nov 01 07:02:41.177002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.176965    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"]
Nov 01 07:02:41.177562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.177545    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" containerName="revoke-keys"
Nov 01 07:02:41.177562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.177562    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" containerName="revoke-keys"
Nov 01 07:02:41.177690 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.177670    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" containerName="revoke-keys"
Nov 01 07:02:41.180143 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.180126    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.234534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.234457    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8e1f56-openshift-default-dockercfg-g4t6l\""
Nov 01 07:02:41.289540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.289506    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"]
Nov 01 07:02:41.291540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.291514    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.292023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292000    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.292263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292242    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjfz\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.292445 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292427    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.292664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292646    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.292836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292815    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.292970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292859    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.293044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.292972    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.293099 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.293050    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394077 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394043    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394085    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjfz\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394115    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394165    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394193    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394228    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394775 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394388    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394775 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394430    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394775 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394545    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.394775 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394580    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.395020 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.394998    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.395210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.395187    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.395335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.395283    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.395591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.395567    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.397171 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.397146    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.397449 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.397430    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.451130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.451065    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjfz\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.451232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.451207    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token\") pod \"e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") " pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.492960 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.492621    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:41.918945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.918920    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"]
Nov 01 07:02:41.919278 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:02:41.919257    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c7f905_1bf0_4d33_9692_81c28ddbc63b.slice/crio-3ea05b6ddd6d53f170eeb7cabe74425e4f3753c7eda2a0ad3b75c32cf9486cb3 WatchSource:0}: Error finding container 3ea05b6ddd6d53f170eeb7cabe74425e4f3753c7eda2a0ad3b75c32cf9486cb3: Status 404 returned error can't find the container with id 3ea05b6ddd6d53f170eeb7cabe74425e4f3753c7eda2a0ad3b75c32cf9486cb3
Nov 01 07:02:41.921855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.921824    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:02:41.921922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.921887    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:02:41.921922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:41.921917    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:02:42.575638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:42.575598    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" event={"ID":"b3c7f905-1bf0-4d33-9692-81c28ddbc63b","Type":"ContainerStarted","Data":"c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7"}
Nov 01 07:02:42.576026 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:42.575678    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" event={"ID":"b3c7f905-1bf0-4d33-9692-81c28ddbc63b","Type":"ContainerStarted","Data":"3ea05b6ddd6d53f170eeb7cabe74425e4f3753c7eda2a0ad3b75c32cf9486cb3"}
Nov 01 07:02:42.742643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:42.742543    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" podStartSLOduration=1.742529372 podStartE2EDuration="1.742529372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:02:42.740645768 +0000 UTC m=+3174.308452595" watchObservedRunningTime="2026-11-01 07:02:42.742529372 +0000 UTC m=+3174.310336202"
Nov 01 07:02:43.493548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:43.493511    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:43.499661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:43.499637    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:43.582636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:43.582603    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:43.583907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:43.583879    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:02:45.289196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.289167    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 07:02:45.450392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450358    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450422    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450504    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450529    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450584    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450605    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450689    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450704    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:02:45.450847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450790    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.450847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450819    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm\") pod \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\" (UID: \"1be60b38-5f66-4a1b-9891-af1ec26fcd24\") "
Nov 01 07:02:45.451090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450867    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:02:45.451090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.450953    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:02:45.451195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451090    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:02:45.451384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451228    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:02:45.451544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451500    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.451544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451520    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.451544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451539    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.451670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451552    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.451670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.451567    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.452922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.452899    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:02:45.453003 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.452985    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:45.453574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.453550    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:02:45.453662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.453586    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm" pod "1be60b38-5f66-4a1b-9891-af1ec26fcd24" (UID: "1be60b38-5f66-4a1b-9891-af1ec26fcd24"). InnerVolumeSpecName "kube-api-access-zhkcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:02:45.552898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.552861    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkcm\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-kube-api-access-zhkcm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.552898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.552895    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.552898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.552907    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.553156 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.552915    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be60b38-5f66-4a1b-9891-af1ec26fcd24-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:02:45.596711 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.596678    4468 generic.go:356] "Generic (PLEG): container finished" podID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerID="bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5" exitCode=0
Nov 01 07:02:45.596906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.596724    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"
Nov 01 07:02:45.596906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.596763    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" event={"ID":"1be60b38-5f66-4a1b-9891-af1ec26fcd24","Type":"ContainerDied","Data":"bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5"}
Nov 01 07:02:45.596906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.596823    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w" event={"ID":"1be60b38-5f66-4a1b-9891-af1ec26fcd24","Type":"ContainerDied","Data":"862f4e5a6211a5508af0fac147b5e92eb8f121c748cc5762e7333835af6087be"}
Nov 01 07:02:45.596906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.596847    4468 scope.go:122] "RemoveContainer" containerID="bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5"
Nov 01 07:02:45.607556 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.607538    4468 scope.go:122] "RemoveContainer" containerID="9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"
Nov 01 07:02:45.617583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.617563    4468 scope.go:122] "RemoveContainer" containerID="bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5"
Nov 01 07:02:45.617838 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:02:45.617819    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5\": container with ID starting with bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5 not found: ID does not exist" containerID="bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5"
Nov 01 07:02:45.617969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.617845    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5"} err="failed to get container status \"bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5\": rpc error: code = NotFound desc = could not find container \"bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5\": container with ID starting with bb0e312927b0b1b944eaf4d1d14e8702519f2496e9e2217a1f7cb56f751d12e5 not found: ID does not exist"
Nov 01 07:02:45.617969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.617868    4468 scope.go:122] "RemoveContainer" containerID="9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"
Nov 01 07:02:45.618084 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:02:45.618074    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d\": container with ID starting with 9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d not found: ID does not exist" containerID="9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"
Nov 01 07:02:45.618138 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.618090    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d"} err="failed to get container status \"9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d\": rpc error: code = NotFound desc = could not find container \"9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d\": container with ID starting with 9fffb8eb4fa400a9c454123799f8d4431a233c2a5f732b806919e8cde291b64d not found: ID does not exist"
Nov 01 07:02:45.761592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.761502    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"]
Nov 01 07:02:45.764297 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:45.764278    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0cb4ef-openshift-default-6d6d6dcb69-4hg4w"]
Nov 01 07:02:46.074066 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074037    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"]
Nov 01 07:02:46.074654 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074639    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy"
Nov 01 07:02:46.074700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074657    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy"
Nov 01 07:02:46.074700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074669    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy"
Nov 01 07:02:46.074700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074675    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy"
Nov 01 07:02:46.074814 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074773    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy"
Nov 01 07:02:46.074814 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.074781    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" containerName="istio-proxy"
Nov 01 07:02:46.077891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.077876    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.131484 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.131431    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8e1f56\"/\"openshift-service-ca.crt\""
Nov 01 07:02:46.132900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.132378    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8e1f56\"/\"kube-root-ca.crt\""
Nov 01 07:02:46.132900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.132510    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8e1f56\"/\"default-dockercfg-7v97c\""
Nov 01 07:02:46.132900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.132577    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8e1f56\"/\"noten-model-8e1f56-kserve-self-signed-certs\""
Nov 01 07:02:46.136658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.136632    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"]
Nov 01 07:02:46.161320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.161284    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.161320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.161317    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.161571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.161539    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.161638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.161605    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjx4\" (UniqueName: \"kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.161731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.161710    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.161790 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.161772    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263319    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjx4\" (UniqueName: \"kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263389    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263427    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263545    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263743 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263563    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263743 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263671    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263896 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263872    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263921    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.263971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.263939    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.265998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.265979    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.266162 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.266145    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.319828 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.319794    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjx4\" (UniqueName: \"kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4\") pod \"noten-model-8e1f56-kserve-896d656c7-8bhkd\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") " pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.389536 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.389440    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:46.590592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.590550    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be60b38-5f66-4a1b-9891-af1ec26fcd24" path="/var/lib/kubelet/pods/1be60b38-5f66-4a1b-9891-af1ec26fcd24/volumes"
Nov 01 07:02:46.764388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:46.764355    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"]
Nov 01 07:02:46.765301 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:02:46.765275    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1195e1fd_2ed5_43cc_8795_a41067b8e299.slice/crio-8554972b20eb95464a4166097e7b05567023bc9dedb21a165ffad4c31d21b408 WatchSource:0}: Error finding container 8554972b20eb95464a4166097e7b05567023bc9dedb21a165ffad4c31d21b408: Status 404 returned error can't find the container with id 8554972b20eb95464a4166097e7b05567023bc9dedb21a165ffad4c31d21b408
Nov 01 07:02:47.612953 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:47.612913    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd" event={"ID":"1195e1fd-2ed5-43cc-8795-a41067b8e299","Type":"ContainerStarted","Data":"c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f"}
Nov 01 07:02:47.613444 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:47.612961    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd" event={"ID":"1195e1fd-2ed5-43cc-8795-a41067b8e299","Type":"ContainerStarted","Data":"8554972b20eb95464a4166097e7b05567023bc9dedb21a165ffad4c31d21b408"}
Nov 01 07:02:47.726211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:47.726160    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd" podStartSLOduration=1.726145512 podStartE2EDuration="1.726145512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:02:47.725545784 +0000 UTC m=+3179.293352624" watchObservedRunningTime="2026-11-01 07:02:47.726145512 +0000 UTC m=+3179.293952342"
Nov 01 07:02:48.389958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:48.389925    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:51.864935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:51.864903    4468 scope.go:122] "RemoveContainer" containerID="60d8e96fe5efcb2fe8246301b4e449faf057d9d22bcea05aa6ccda71912ba83e"
Nov 01 07:02:56.390433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:56.390401    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:56.404366 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:56.404341    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:56.678927 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:56.678850    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:02:58.464709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:02:58.464664    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:03:02.947071 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:02.946981    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:03:05.134428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.134393    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-254a3e"]
Nov 01 07:03:05.139736 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.139715    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:05.188453 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.188425    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6kv\" (UniqueName: \"kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv\") pod \"test-curl-7686-254a3e\" (UID: \"222e831b-7517-46e7-b3cc-81067785aee6\") " pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:05.193853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.193825    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-254a3e"]
Nov 01 07:03:05.239567 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.239532    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\""
Nov 01 07:03:05.289283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.289237    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6kv\" (UniqueName: \"kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv\") pod \"test-curl-7686-254a3e\" (UID: \"222e831b-7517-46e7-b3cc-81067785aee6\") " pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:05.345553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.345524    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6kv\" (UniqueName: \"kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv\") pod \"test-curl-7686-254a3e\" (UID: \"222e831b-7517-46e7-b3cc-81067785aee6\") " pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:05.452108 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.452000    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:05.826770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:05.826741    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-254a3e"]
Nov 01 07:03:05.828440 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:05.828409    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod222e831b_7517_46e7_b3cc_81067785aee6.slice/crio-e776129baa075ffb18c56179af07173bee823e075d09b52deddc971956089287 WatchSource:0}: Error finding container e776129baa075ffb18c56179af07173bee823e075d09b52deddc971956089287: Status 404 returned error can't find the container with id e776129baa075ffb18c56179af07173bee823e075d09b52deddc971956089287
Nov 01 07:03:06.733548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:06.733504    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-254a3e" event={"ID":"222e831b-7517-46e7-b3cc-81067785aee6","Type":"ContainerStarted","Data":"49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4"}
Nov 01 07:03:06.733548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:06.733552    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-254a3e" event={"ID":"222e831b-7517-46e7-b3cc-81067785aee6","Type":"ContainerStarted","Data":"e776129baa075ffb18c56179af07173bee823e075d09b52deddc971956089287"}
Nov 01 07:03:06.847290 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:06.847241    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-254a3e" podStartSLOduration=1.847228697 podStartE2EDuration="1.847228697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:06.844248896 +0000 UTC m=+3198.412055726" watchObservedRunningTime="2026-11-01 07:03:06.847228697 +0000 UTC m=+3198.415035526"
Nov 01 07:03:07.649068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:07.649034    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"]
Nov 01 07:03:07.649352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:07.649326    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy" containerID="cri-o://73ae624fec91da445608d3507fba0b690cdab660efb07f5ef8273751505b2a86" gracePeriod=30
Nov 01 07:03:08.235773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.235734    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-254a3e"]
Nov 01 07:03:08.240364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.240318    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-254a3e"]
Nov 01 07:03:08.393403 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.393363    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-3f6539"]
Nov 01 07:03:08.394011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.393996    4468 cpu_manager.go:414] "containerMap: removing container" podUID="222e831b-7517-46e7-b3cc-81067785aee6" containerName="test-curl-7686-254a3e"
Nov 01 07:03:08.394064 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.394013    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222e831b-7517-46e7-b3cc-81067785aee6" containerName="test-curl-7686-254a3e"
Nov 01 07:03:08.394134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.394123    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="222e831b-7517-46e7-b3cc-81067785aee6" containerName="test-curl-7686-254a3e"
Nov 01 07:03:08.397641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.397625    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:08.452293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.452265    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-3f6539"]
Nov 01 07:03:08.528545 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.528434    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps75r\" (UniqueName: \"kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r\") pod \"test-curl-7686-3f6539\" (UID: \"c479f50b-eef2-4b5e-b224-aa89428efdce\") " pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:08.630219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.630183    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps75r\" (UniqueName: \"kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r\") pod \"test-curl-7686-3f6539\" (UID: \"c479f50b-eef2-4b5e-b224-aa89428efdce\") " pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:08.686239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.686209    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps75r\" (UniqueName: \"kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r\") pod \"test-curl-7686-3f6539\" (UID: \"c479f50b-eef2-4b5e-b224-aa89428efdce\") " pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:08.710213 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.710190    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:08.747346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:08.747310    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-254a3e" podUID="222e831b-7517-46e7-b3cc-81067785aee6" containerName="test-curl-7686-254a3e" containerID="cri-o://49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4" gracePeriod=2
Nov 01 07:03:09.086357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:09.086322    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-3f6539"]
Nov 01 07:03:09.087399 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:09.087374    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc479f50b_eef2_4b5e_b224_aa89428efdce.slice/crio-57119036c7854c860dc70888894bebaac30049abe99145224c795c0449800441 WatchSource:0}: Error finding container 57119036c7854c860dc70888894bebaac30049abe99145224c795c0449800441: Status 404 returned error can't find the container with id 57119036c7854c860dc70888894bebaac30049abe99145224c795c0449800441
Nov 01 07:03:09.755047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:09.755011    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-3f6539" event={"ID":"c479f50b-eef2-4b5e-b224-aa89428efdce","Type":"ContainerStarted","Data":"3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a"}
Nov 01 07:03:09.755581 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:09.755059    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-3f6539" event={"ID":"c479f50b-eef2-4b5e-b224-aa89428efdce","Type":"ContainerStarted","Data":"57119036c7854c860dc70888894bebaac30049abe99145224c795c0449800441"}
Nov 01 07:03:09.866969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:09.866904    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-3f6539" podStartSLOduration=1.866889115 podStartE2EDuration="1.866889115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:09.865439654 +0000 UTC m=+3201.433246485" watchObservedRunningTime="2026-11-01 07:03:09.866889115 +0000 UTC m=+3201.434695944"
Nov 01 07:03:11.022235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.022174    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:11.162209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.162183    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv\" (UniqueName: \"kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv\") pod \"222e831b-7517-46e7-b3cc-81067785aee6\" (UID: \"222e831b-7517-46e7-b3cc-81067785aee6\") "
Nov 01 07:03:11.164337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.164312    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv" pod "222e831b-7517-46e7-b3cc-81067785aee6" (UID: "222e831b-7517-46e7-b3cc-81067785aee6"). InnerVolumeSpecName "kube-api-access-dm6kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:11.247051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.247025    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-3f6539"]
Nov 01 07:03:11.251149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.251124    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-3f6539"]
Nov 01 07:03:11.263696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.263660    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm6kv\" (UniqueName: \"kubernetes.io/projected/222e831b-7517-46e7-b3cc-81067785aee6-kube-api-access-dm6kv\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:11.770163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.770115    4468 generic.go:356] "Generic (PLEG): container finished" podID="222e831b-7517-46e7-b3cc-81067785aee6" containerID="49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4" exitCode=137
Nov 01 07:03:11.770380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.770169    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:11.770380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.770184    4468 scope.go:122] "RemoveContainer" containerID="49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4"
Nov 01 07:03:11.770547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.770483    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-3f6539" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" containerName="test-curl-7686-3f6539" containerID="cri-o://3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a" gracePeriod=2
Nov 01 07:03:11.779805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.779783    4468 scope.go:122] "RemoveContainer" containerID="49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4"
Nov 01 07:03:11.780117 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:11.780096    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4\": container with ID starting with 49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4 not found: ID does not exist" containerID="49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4"
Nov 01 07:03:11.780181 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:11.780126    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4"} err="failed to get container status \"49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4\": rpc error: code = NotFound desc = could not find container \"49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4\": container with ID starting with 49ccf1fa196d1e8dfc5e4301a19ee379bd10f4ee2434be73730190dcffa06db4 not found: ID does not exist"
Nov 01 07:03:11.821612 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:11.821567    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3f6539\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:11.872486 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:11.872439    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3f6539\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:11.923037 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:11.923002    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-254a3e\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="222e831b-7517-46e7-b3cc-81067785aee6" pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:12.273891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.273854    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"]
Nov 01 07:03:12.274614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.274592    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" containerName="test-curl-7686-3f6539"
Nov 01 07:03:12.274735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.274614    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" containerName="test-curl-7686-3f6539"
Nov 01 07:03:12.274818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.274804    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" containerName="test-curl-7686-3f6539"
Nov 01 07:03:12.279183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.279159    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.330777 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:12.330737    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-254a3e\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="222e831b-7517-46e7-b3cc-81067785aee6" pod="opendatahub/test-curl-7686-254a3e"
Nov 01 07:03:12.331605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.331571    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:12.334338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.334312    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"]
Nov 01 07:03:12.375836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.375799    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.376007 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.375848    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq75\" (UniqueName: \"kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.376167 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.376143    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.379576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.379551    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:12.379710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.379677    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:03:12.379710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.379678    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:12.477088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.477053    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq75\" (UniqueName: \"kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.477276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.477257    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.477395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.477375    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.478081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.478054    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.480220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.480188    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.492023 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:12.491992    4468 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3f6539\" is forbidden: User \"system:node:ip-10-0-138-21.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-21.us-west-1.compute.internal' and this object" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:12.533604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.533535    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq75\" (UniqueName: \"kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75\") pod \"maas-api-revoke-sub-da1160c6-ktpqb\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.590676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.590641    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222e831b-7517-46e7-b3cc-81067785aee6" path="/var/lib/kubelet/pods/222e831b-7517-46e7-b3cc-81067785aee6/volumes"
Nov 01 07:03:12.592515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.592493    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:12.779010 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.778982    4468 generic.go:356] "Generic (PLEG): container finished" podID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerID="73ae624fec91da445608d3507fba0b690cdab660efb07f5ef8273751505b2a86" exitCode=0
Nov 01 07:03:12.779199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.779059    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" event={"ID":"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23","Type":"ContainerDied","Data":"73ae624fec91da445608d3507fba0b690cdab660efb07f5ef8273751505b2a86"}
Nov 01 07:03:12.779199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.779132    4468 scope.go:122] "RemoveContainer" containerID="3c46bdace1549578bb13ebf9ae0188c1ad7d37eae0670a8440e1b1e19f6d13b9"
Nov 01 07:03:12.897101 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.897075    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 07:03:12.968169 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:12.968140    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca52439_a747_4533_b4c7_9c04c77ca8cd.slice/crio-0f27a4e32ca044b257efe35ff926eb4b5596d17f6ea2db6612d0bdcad4d012cc WatchSource:0}: Error finding container 0f27a4e32ca044b257efe35ff926eb4b5596d17f6ea2db6612d0bdcad4d012cc: Status 404 returned error can't find the container with id 0f27a4e32ca044b257efe35ff926eb4b5596d17f6ea2db6612d0bdcad4d012cc
Nov 01 07:03:12.984432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984410    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.984538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984520    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.984612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984567    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.984666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984610    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.984666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984652    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.984805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984775    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:12.984924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984811    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.984924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984889    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:12.984924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984901    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:12.985112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.984950    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.985112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985026    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.985112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985094    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token\") pod \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\" (UID: \"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23\") "
Nov 01 07:03:12.985304 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985263    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:12.985304 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985297    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:12.985572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985550    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:12.985669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985572    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:12.985669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985587    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:12.985669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985599    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:12.985669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.985612    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:12.987300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.987279    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:03:12.987716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.987698    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:12.987781 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.987721    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:12.987781 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:12.987741    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6" pod "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" (UID: "0f3e1cbd-bfea-4a36-8e99-52507f1c8b23"). InnerVolumeSpecName "kube-api-access-6cpp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:13.002942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.002911    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"]
Nov 01 07:03:13.087401 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.087369    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpp6\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-kube-api-access-6cpp6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:13.087401 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.087396    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:13.087401 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.087407    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:13.087671 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.087415    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:13.350120 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.350035    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"]
Nov 01 07:03:13.505543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.505460    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"]
Nov 01 07:03:13.505783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.505723    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" podUID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" containerName="istio-proxy" containerID="cri-o://c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7" gracePeriod=30
Nov 01 07:03:13.585927 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.585886    4468 patch_prober.go:29] interesting pod/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:03:13.586179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.585953    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" podUID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:03:13.786422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.786397    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"
Nov 01 07:03:13.786693 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.786439    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5" event={"ID":"0f3e1cbd-bfea-4a36-8e99-52507f1c8b23","Type":"ContainerDied","Data":"7e17f5f79e393bffb85453cde8dbf75383654c17f6075b88f583ad5499f45cc9"}
Nov 01 07:03:13.786693 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.786520    4468 scope.go:122] "RemoveContainer" containerID="73ae624fec91da445608d3507fba0b690cdab660efb07f5ef8273751505b2a86"
Nov 01 07:03:13.788409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.788379    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb" event={"ID":"aca52439-a747-4533-b4c7-9c04c77ca8cd","Type":"ContainerStarted","Data":"3fa1a9b0b3f244edcbf63e5368f9d51dd2d16b4f134a09c1ff49c8b2b5806aa3"}
Nov 01 07:03:13.788541 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.788439    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb" event={"ID":"aca52439-a747-4533-b4c7-9c04c77ca8cd","Type":"ContainerStarted","Data":"0f27a4e32ca044b257efe35ff926eb4b5596d17f6ea2db6612d0bdcad4d012cc"}
Nov 01 07:03:13.953088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:13.953033    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb" podStartSLOduration=1.9530178980000001 podStartE2EDuration="1.953017898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:13.951499109 +0000 UTC m=+3205.519305938" watchObservedRunningTime="2026-11-01 07:03:13.953017898 +0000 UTC m=+3205.520824726"
Nov 01 07:03:14.023871 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.023837    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:14.067412 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.067355    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"]
Nov 01 07:03:14.070357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.070328    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-hsvw5"]
Nov 01 07:03:14.103851 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.103815    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r\" (UniqueName: \"kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r\") pod \"c479f50b-eef2-4b5e-b224-aa89428efdce\" (UID: \"c479f50b-eef2-4b5e-b224-aa89428efdce\") "
Nov 01 07:03:14.106151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.106121    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r" pod "c479f50b-eef2-4b5e-b224-aa89428efdce" (UID: "c479f50b-eef2-4b5e-b224-aa89428efdce"). InnerVolumeSpecName "kube-api-access-ps75r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:14.205954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.205919    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps75r\" (UniqueName: \"kubernetes.io/projected/c479f50b-eef2-4b5e-b224-aa89428efdce-kube-api-access-ps75r\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:14.590332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.590299    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" path="/var/lib/kubelet/pods/0f3e1cbd-bfea-4a36-8e99-52507f1c8b23/volumes"
Nov 01 07:03:14.590960 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.590940    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c479f50b-eef2-4b5e-b224-aa89428efdce" path="/var/lib/kubelet/pods/c479f50b-eef2-4b5e-b224-aa89428efdce/volumes"
Nov 01 07:03:14.797688 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.797601    4468 generic.go:356] "Generic (PLEG): container finished" podID="c479f50b-eef2-4b5e-b224-aa89428efdce" containerID="3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a" exitCode=137
Nov 01 07:03:14.797688 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.797645    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3f6539"
Nov 01 07:03:14.797916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.797697    4468 scope.go:122] "RemoveContainer" containerID="3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a"
Nov 01 07:03:14.800081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.800053    4468 generic.go:356] "Generic (PLEG): container finished" podID="aca52439-a747-4533-b4c7-9c04c77ca8cd" containerID="3fa1a9b0b3f244edcbf63e5368f9d51dd2d16b4f134a09c1ff49c8b2b5806aa3" exitCode=0
Nov 01 07:03:14.800202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.800092    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb" event={"ID":"aca52439-a747-4533-b4c7-9c04c77ca8cd","Type":"ContainerDied","Data":"3fa1a9b0b3f244edcbf63e5368f9d51dd2d16b4f134a09c1ff49c8b2b5806aa3"}
Nov 01 07:03:14.807349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.807329    4468 scope.go:122] "RemoveContainer" containerID="3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a"
Nov 01 07:03:14.807684 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:14.807659    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a\": container with ID starting with 3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a not found: ID does not exist" containerID="3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a"
Nov 01 07:03:14.807784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:14.807695    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a"} err="failed to get container status \"3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a\": rpc error: code = NotFound desc = could not find container \"3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a\": container with ID starting with 3c52cd65f0334ae4c640ba9d30a9f0727ca6794b08e16a18f8a5abc53f17bc7a not found: ID does not exist"
Nov 01 07:03:15.964339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:15.964313    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:16.131881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.131801    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75\" (UniqueName: \"kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75\") pod \"aca52439-a747-4533-b4c7-9c04c77ca8cd\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") "
Nov 01 07:03:16.131881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.131843    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca\") pod \"aca52439-a747-4533-b4c7-9c04c77ca8cd\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") "
Nov 01 07:03:16.132074 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.131939    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca\") pod \"aca52439-a747-4533-b4c7-9c04c77ca8cd\" (UID: \"aca52439-a747-4533-b4c7-9c04c77ca8cd\") "
Nov 01 07:03:16.132284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.132249    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca" pod "aca52439-a747-4533-b4c7-9c04c77ca8cd" (UID: "aca52439-a747-4533-b4c7-9c04c77ca8cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:16.132780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.132758    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:16.134822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.134797    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca" pod "aca52439-a747-4533-b4c7-9c04c77ca8cd" (UID: "aca52439-a747-4533-b4c7-9c04c77ca8cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:16.134917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.134840    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75" pod "aca52439-a747-4533-b4c7-9c04c77ca8cd" (UID: "aca52439-a747-4533-b4c7-9c04c77ca8cd"). InnerVolumeSpecName "kube-api-access-rkq75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:16.236477 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.236441    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkq75\" (UniqueName: \"kubernetes.io/projected/aca52439-a747-4533-b4c7-9c04c77ca8cd-kube-api-access-rkq75\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:16.236671 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.236487    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca52439-a747-4533-b4c7-9c04c77ca8cd-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:16.815885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.815843    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb" event={"ID":"aca52439-a747-4533-b4c7-9c04c77ca8cd","Type":"ContainerDied","Data":"0f27a4e32ca044b257efe35ff926eb4b5596d17f6ea2db6612d0bdcad4d012cc"}
Nov 01 07:03:16.815885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.815872    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"
Nov 01 07:03:16.816110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:16.815909    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f27a4e32ca044b257efe35ff926eb4b5596d17f6ea2db6612d0bdcad4d012cc"
Nov 01 07:03:18.798954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.798924    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:03:18.830045 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.830012    4468 generic.go:356] "Generic (PLEG): container finished" podID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" containerID="c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7" exitCode=0
Nov 01 07:03:18.830225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.830069    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"
Nov 01 07:03:18.830225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.830089    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" event={"ID":"b3c7f905-1bf0-4d33-9692-81c28ddbc63b","Type":"ContainerDied","Data":"c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7"}
Nov 01 07:03:18.830225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.830143    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw" event={"ID":"b3c7f905-1bf0-4d33-9692-81c28ddbc63b","Type":"ContainerDied","Data":"3ea05b6ddd6d53f170eeb7cabe74425e4f3753c7eda2a0ad3b75c32cf9486cb3"}
Nov 01 07:03:18.830225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.830165    4468 scope.go:122] "RemoveContainer" containerID="c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7"
Nov 01 07:03:18.840060 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.840041    4468 scope.go:122] "RemoveContainer" containerID="c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7"
Nov 01 07:03:18.840334 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:18.840317    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7\": container with ID starting with c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7 not found: ID does not exist" containerID="c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7"
Nov 01 07:03:18.840414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.840342    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7"} err="failed to get container status \"c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7\": rpc error: code = NotFound desc = could not find container \"c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7\": container with ID starting with c2b7ae6ac686c926b129f664a1d193631133696971954a75f59f92176563b0c7 not found: ID does not exist"
Nov 01 07:03:18.969974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.969940    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.969995    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970022    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970068    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970124    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970206    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970269    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970310    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970345    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy\") pod \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\" (UID: \"b3c7f905-1bf0-4d33-9692-81c28ddbc63b\") "
Nov 01 07:03:18.970406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970391    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:18.970742 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970428    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:18.970742 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970649    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:18.970742 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.970707    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:18.971225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.971004    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:18.971225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.971021    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:18.971225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.971028    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:18.971225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.971065    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:18.971225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.971083    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:18.973459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.973431    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:03:18.973614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.973516    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "kube-api-access-6kjfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:18.973685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.973621    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:18.973764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:18.973739    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token" pod "b3c7f905-1bf0-4d33-9692-81c28ddbc63b" (UID: "b3c7f905-1bf0-4d33-9692-81c28ddbc63b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:19.072770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.072686    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:19.072770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.072716    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjfz\" (UniqueName: \"kubernetes.io/projected/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-kube-api-access-6kjfz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:19.072770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.072728    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:19.072770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.072739    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:19.072770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.072747    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c7f905-1bf0-4d33-9692-81c28ddbc63b-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:19.252206 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.252163    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"]
Nov 01 07:03:19.256116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:19.256087    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8e1f56-openshift-default-64bf794c5f-55kxw"]
Nov 01 07:03:20.590419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:20.590383    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" path="/var/lib/kubelet/pods/b3c7f905-1bf0-4d33-9692-81c28ddbc63b/volumes"
Nov 01 07:03:23.062040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.061993    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"]
Nov 01 07:03:23.063956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.063927    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064132    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064177    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064186    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064236    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" containerName="istio-proxy"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064244    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" containerName="istio-proxy"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064253    4468 cpu_manager.go:414] "containerMap: removing container" podUID="aca52439-a747-4533-b4c7-9c04c77ca8cd" containerName="revoke-keys"
Nov 01 07:03:23.064392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064262    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca52439-a747-4533-b4c7-9c04c77ca8cd" containerName="revoke-keys"
Nov 01 07:03:23.071118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064517    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy"
Nov 01 07:03:23.071118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064546    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c7f905-1bf0-4d33-9692-81c28ddbc63b" containerName="istio-proxy"
Nov 01 07:03:23.071118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.064555    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca52439-a747-4533-b4c7-9c04c77ca8cd" containerName="revoke-keys"
Nov 01 07:03:23.071695 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.071668    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.123029 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.122996    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"]
Nov 01 07:03:23.169487 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.169436    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:23.169487 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.169486    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:03:23.169696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.169496    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:23.173859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.173835    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:23.222839 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.222806    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2k66\" (UniqueName: \"kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.223034 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.222929    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.223034 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.223020    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.324689 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.324601    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.324873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.324692    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2k66\" (UniqueName: \"kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.324873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.324751    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.325309 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.325282    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.327192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.327172    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.381036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.380998    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2k66\" (UniqueName: \"kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66\") pod \"maas-api-revoke-sub-741cad88-m699g\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.388931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.388905    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:23.770521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.770481    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"]
Nov 01 07:03:23.772130 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:23.772104    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a47f97_ca09_46af_b4a4_979e27ed3947.slice/crio-1e67082b990fb0e594deae9f1074a39d9119d857588730a353ed65b8dea3f652 WatchSource:0}: Error finding container 1e67082b990fb0e594deae9f1074a39d9119d857588730a353ed65b8dea3f652: Status 404 returned error can't find the container with id 1e67082b990fb0e594deae9f1074a39d9119d857588730a353ed65b8dea3f652
Nov 01 07:03:23.869621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:23.869591    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g" event={"ID":"e7a47f97-ca09-46af-b4a4-979e27ed3947","Type":"ContainerStarted","Data":"1e67082b990fb0e594deae9f1074a39d9119d857588730a353ed65b8dea3f652"}
Nov 01 07:03:24.877511 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:24.877479    4468 generic.go:356] "Generic (PLEG): container finished" podID="e7a47f97-ca09-46af-b4a4-979e27ed3947" containerID="10e6bb87223ec84b1b2e44356619bf3183a6e438e555607e37ef0b9ffa67522f" exitCode=0
Nov 01 07:03:24.877911 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:24.877558    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g" event={"ID":"e7a47f97-ca09-46af-b4a4-979e27ed3947","Type":"ContainerDied","Data":"10e6bb87223ec84b1b2e44356619bf3183a6e438e555607e37ef0b9ffa67522f"}
Nov 01 07:03:26.038275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.038249    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:26.057811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.057781    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca\") pod \"e7a47f97-ca09-46af-b4a4-979e27ed3947\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") "
Nov 01 07:03:26.057950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.057920    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66\" (UniqueName: \"kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66\") pod \"e7a47f97-ca09-46af-b4a4-979e27ed3947\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") "
Nov 01 07:03:26.058211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.057982    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca\") pod \"e7a47f97-ca09-46af-b4a4-979e27ed3947\" (UID: \"e7a47f97-ca09-46af-b4a4-979e27ed3947\") "
Nov 01 07:03:26.058363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.058344    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca" pod "e7a47f97-ca09-46af-b4a4-979e27ed3947" (UID: "e7a47f97-ca09-46af-b4a4-979e27ed3947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:26.060138 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.060113    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66" pod "e7a47f97-ca09-46af-b4a4-979e27ed3947" (UID: "e7a47f97-ca09-46af-b4a4-979e27ed3947"). InnerVolumeSpecName "kube-api-access-p2k66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:26.060289 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.060271    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca" pod "e7a47f97-ca09-46af-b4a4-979e27ed3947" (UID: "e7a47f97-ca09-46af-b4a4-979e27ed3947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:26.160061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.159984    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:26.160061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.160017    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2k66\" (UniqueName: \"kubernetes.io/projected/e7a47f97-ca09-46af-b4a4-979e27ed3947-kube-api-access-p2k66\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:26.160061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.160031    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a47f97-ca09-46af-b4a4-979e27ed3947-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:26.891849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.891762    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"
Nov 01 07:03:26.891849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.891796    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g" event={"ID":"e7a47f97-ca09-46af-b4a4-979e27ed3947","Type":"ContainerDied","Data":"1e67082b990fb0e594deae9f1074a39d9119d857588730a353ed65b8dea3f652"}
Nov 01 07:03:26.891849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:26.891842    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e67082b990fb0e594deae9f1074a39d9119d857588730a353ed65b8dea3f652"
Nov 01 07:03:28.370610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.370542    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd" podUID="1195e1fd-2ed5-43cc-8795-a41067b8e299" containerName="main" containerID="cri-o://c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f" gracePeriod=45
Nov 01 07:03:28.638747 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.638688    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:03:28.701052 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701022    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs\" (UniqueName: \"kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs\") pod \"1195e1fd-2ed5-43cc-8795-a41067b8e299\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") "
Nov 01 07:03:28.701232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701066    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home\") pod \"1195e1fd-2ed5-43cc-8795-a41067b8e299\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") "
Nov 01 07:03:28.701232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701135    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir\") pod \"1195e1fd-2ed5-43cc-8795-a41067b8e299\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") "
Nov 01 07:03:28.701345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701304    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache\") pod \"1195e1fd-2ed5-43cc-8795-a41067b8e299\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") "
Nov 01 07:03:28.701409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701345    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4\" (UniqueName: \"kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4\") pod \"1195e1fd-2ed5-43cc-8795-a41067b8e299\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") "
Nov 01 07:03:28.701409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701361    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home" pod "1195e1fd-2ed5-43cc-8795-a41067b8e299" (UID: "1195e1fd-2ed5-43cc-8795-a41067b8e299"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:28.701409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701380    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm\") pod \"1195e1fd-2ed5-43cc-8795-a41067b8e299\" (UID: \"1195e1fd-2ed5-43cc-8795-a41067b8e299\") "
Nov 01 07:03:28.701409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701386    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir" pod "1195e1fd-2ed5-43cc-8795-a41067b8e299" (UID: "1195e1fd-2ed5-43cc-8795-a41067b8e299"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:28.701650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701586    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache" pod "1195e1fd-2ed5-43cc-8795-a41067b8e299" (UID: "1195e1fd-2ed5-43cc-8795-a41067b8e299"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:28.702016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.701993    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:28.702201 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.702175    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:28.702318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.702204    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:28.703505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.703481    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs" pod "1195e1fd-2ed5-43cc-8795-a41067b8e299" (UID: "1195e1fd-2ed5-43cc-8795-a41067b8e299"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:28.703610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.703591    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm" pod "1195e1fd-2ed5-43cc-8795-a41067b8e299" (UID: "1195e1fd-2ed5-43cc-8795-a41067b8e299"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:28.703777 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.703757    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4" pod "1195e1fd-2ed5-43cc-8795-a41067b8e299" (UID: "1195e1fd-2ed5-43cc-8795-a41067b8e299"). InnerVolumeSpecName "kube-api-access-tpjx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:28.803320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.803284    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjx4\" (UniqueName: \"kubernetes.io/projected/1195e1fd-2ed5-43cc-8795-a41067b8e299-kube-api-access-tpjx4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:28.803320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.803315    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1195e1fd-2ed5-43cc-8795-a41067b8e299-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:28.803320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.803325    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1195e1fd-2ed5-43cc-8795-a41067b8e299-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:28.906668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.906566    4468 generic.go:356] "Generic (PLEG): container finished" podID="1195e1fd-2ed5-43cc-8795-a41067b8e299" containerID="c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f" exitCode=0
Nov 01 07:03:28.906837 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.906675    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"
Nov 01 07:03:28.906837 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.906720    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd" event={"ID":"1195e1fd-2ed5-43cc-8795-a41067b8e299","Type":"ContainerDied","Data":"c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f"}
Nov 01 07:03:28.906837 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.906763    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd" event={"ID":"1195e1fd-2ed5-43cc-8795-a41067b8e299","Type":"ContainerDied","Data":"8554972b20eb95464a4166097e7b05567023bc9dedb21a165ffad4c31d21b408"}
Nov 01 07:03:28.906837 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.906778    4468 scope.go:122] "RemoveContainer" containerID="c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f"
Nov 01 07:03:28.917480 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.917419    4468 scope.go:122] "RemoveContainer" containerID="c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f"
Nov 01 07:03:28.917723 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:28.917706    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f\": container with ID starting with c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f not found: ID does not exist" containerID="c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f"
Nov 01 07:03:28.917779 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:28.917730    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f"} err="failed to get container status \"c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f\": rpc error: code = NotFound desc = could not find container \"c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f\": container with ID starting with c0e0dfe092502806cd82127520adfe3f11018170eabc5395e1f7ee9506ec674f not found: ID does not exist"
Nov 01 07:03:29.069998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:29.069954    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"]
Nov 01 07:03:29.072896 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:29.072875    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8e1f56/noten-model-8e1f56-kserve-896d656c7-8bhkd"]
Nov 01 07:03:30.590697 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:30.590660    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1195e1fd-2ed5-43cc-8795-a41067b8e299" path="/var/lib/kubelet/pods/1195e1fd-2ed5-43cc-8795-a41067b8e299/volumes"
Nov 01 07:03:32.946776 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:32.946740    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:03:34.823710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.823668    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"]
Nov 01 07:03:34.824381 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824359    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e7a47f97-ca09-46af-b4a4-979e27ed3947" containerName="revoke-keys"
Nov 01 07:03:34.824500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824383    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a47f97-ca09-46af-b4a4-979e27ed3947" containerName="revoke-keys"
Nov 01 07:03:34.824500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824403    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1195e1fd-2ed5-43cc-8795-a41067b8e299" containerName="main"
Nov 01 07:03:34.824500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824411    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195e1fd-2ed5-43cc-8795-a41067b8e299" containerName="main"
Nov 01 07:03:34.824664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824640    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a47f97-ca09-46af-b4a4-979e27ed3947" containerName="revoke-keys"
Nov 01 07:03:34.824664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824654    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195e1fd-2ed5-43cc-8795-a41067b8e299" containerName="main"
Nov 01 07:03:34.824764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.824668    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3e1cbd-bfea-4a36-8e99-52507f1c8b23" containerName="istio-proxy"
Nov 01 07:03:34.829100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.829079    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.874168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.874132    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.874324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.874180    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.874324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.874310    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvskc\" (UniqueName: \"kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.883185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.883155    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"]
Nov 01 07:03:34.928902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.928873    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:34.929074 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.928917    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:34.929074 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.928917    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:34.930606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.930589    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:03:34.975933 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.975900    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.976168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.975948    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.976168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.976014    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvskc\" (UniqueName: \"kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.976605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.976585    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:34.978454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:34.978433    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:35.032690 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:35.032659    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvskc\" (UniqueName: \"kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc\") pod \"maas-api-revoke-sub-b5abfaf6-7h7ct\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:35.142056 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:35.141975    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:35.524210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:35.524180    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"]
Nov 01 07:03:35.525349 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:35.525322    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3840bf_184f_401e_8fb9_52aaf400a0ec.slice/crio-3d0858eb39b30f847a88579a6fdf77bae075cacff44495ddaf85fa4a4cbe87e1 WatchSource:0}: Error finding container 3d0858eb39b30f847a88579a6fdf77bae075cacff44495ddaf85fa4a4cbe87e1: Status 404 returned error can't find the container with id 3d0858eb39b30f847a88579a6fdf77bae075cacff44495ddaf85fa4a4cbe87e1
Nov 01 07:03:35.955539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:35.955494    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct" event={"ID":"9d3840bf-184f-401e-8fb9-52aaf400a0ec","Type":"ContainerStarted","Data":"88498cb4145dbb1794fb4974a040a60941c2d538a68a88996c2d7271b6d8ec15"}
Nov 01 07:03:35.956019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:35.955552    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct" event={"ID":"9d3840bf-184f-401e-8fb9-52aaf400a0ec","Type":"ContainerStarted","Data":"3d0858eb39b30f847a88579a6fdf77bae075cacff44495ddaf85fa4a4cbe87e1"}
Nov 01 07:03:36.068897 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:36.068782    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct" podStartSLOduration=2.068763505 podStartE2EDuration="2.068763505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:36.067039362 +0000 UTC m=+3227.634846189" watchObservedRunningTime="2026-11-01 07:03:36.068763505 +0000 UTC m=+3227.636570335"
Nov 01 07:03:36.963304 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:36.963269    4468 generic.go:356] "Generic (PLEG): container finished" podID="9d3840bf-184f-401e-8fb9-52aaf400a0ec" containerID="88498cb4145dbb1794fb4974a040a60941c2d538a68a88996c2d7271b6d8ec15" exitCode=0
Nov 01 07:03:36.963786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:36.963355    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct" event={"ID":"9d3840bf-184f-401e-8fb9-52aaf400a0ec","Type":"ContainerDied","Data":"88498cb4145dbb1794fb4974a040a60941c2d538a68a88996c2d7271b6d8ec15"}
Nov 01 07:03:38.119491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.119434    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:38.220298 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.220208    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc\" (UniqueName: \"kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc\") pod \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") "
Nov 01 07:03:38.220298 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.220272    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca\") pod \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") "
Nov 01 07:03:38.220589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.220305    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca\") pod \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\" (UID: \"9d3840bf-184f-401e-8fb9-52aaf400a0ec\") "
Nov 01 07:03:38.220731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.220707    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca" pod "9d3840bf-184f-401e-8fb9-52aaf400a0ec" (UID: "9d3840bf-184f-401e-8fb9-52aaf400a0ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:38.222506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.222461    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca" pod "9d3840bf-184f-401e-8fb9-52aaf400a0ec" (UID: "9d3840bf-184f-401e-8fb9-52aaf400a0ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:38.222506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.222496    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc" pod "9d3840bf-184f-401e-8fb9-52aaf400a0ec" (UID: "9d3840bf-184f-401e-8fb9-52aaf400a0ec"). InnerVolumeSpecName "kube-api-access-bvskc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:38.322353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.322315    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvskc\" (UniqueName: \"kubernetes.io/projected/9d3840bf-184f-401e-8fb9-52aaf400a0ec-kube-api-access-bvskc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:38.322353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.322346    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:38.322353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.322356    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3840bf-184f-401e-8fb9-52aaf400a0ec-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:38.977076 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.976988    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"
Nov 01 07:03:38.977076 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.977017    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct" event={"ID":"9d3840bf-184f-401e-8fb9-52aaf400a0ec","Type":"ContainerDied","Data":"3d0858eb39b30f847a88579a6fdf77bae075cacff44495ddaf85fa4a4cbe87e1"}
Nov 01 07:03:38.977076 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:38.977072    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0858eb39b30f847a88579a6fdf77bae075cacff44495ddaf85fa4a4cbe87e1"
Nov 01 07:03:39.233809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:39.233726    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"]
Nov 01 07:03:39.234255 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:39.233988    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy" containerID="cri-o://835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3" gracePeriod=30
Nov 01 07:03:44.506866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.506801    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608141    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608244    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608299    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608371    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608408    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608434    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608457    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608567    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608613    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data\") pod \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\" (UID: \"b24aeff4-8c1e-4638-ae19-8c09010af0f6\") "
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.608851    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.609064    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:44.609073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.609066    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:44.609835 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.609127    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.609835 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.609218    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:44.609835 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.609543    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:44.611380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.611349    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:44.611380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.611362    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "kube-api-access-6hgm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:44.611797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.611775    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:03:44.612015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.611994    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy" pod "b24aeff4-8c1e-4638-ae19-8c09010af0f6" (UID: "b24aeff4-8c1e-4638-ae19-8c09010af0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:03:44.710293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710257    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgm8\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-kube-api-access-6hgm8\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710290    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710300    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710309    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710318    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710325    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710333    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:44.710580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:44.710341    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24aeff4-8c1e-4638-ae19-8c09010af0f6-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:45.031390 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.031356    4468 generic.go:356] "Generic (PLEG): container finished" podID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerID="835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3" exitCode=0
Nov 01 07:03:45.031585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.031401    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"
Nov 01 07:03:45.031585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.031424    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" event={"ID":"b24aeff4-8c1e-4638-ae19-8c09010af0f6","Type":"ContainerDied","Data":"835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3"}
Nov 01 07:03:45.031585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.031490    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp" event={"ID":"b24aeff4-8c1e-4638-ae19-8c09010af0f6","Type":"ContainerDied","Data":"4c2ea8e0ae056e2ba0a067a1a8ae4e6666a252aadd31df2fc718935f5b672685"}
Nov 01 07:03:45.031585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.031516    4468 scope.go:122] "RemoveContainer" containerID="835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3"
Nov 01 07:03:45.042527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.042500    4468 scope.go:122] "RemoveContainer" containerID="8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"
Nov 01 07:03:45.051732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.051710    4468 scope.go:122] "RemoveContainer" containerID="835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3"
Nov 01 07:03:45.052009 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:45.051990    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3\": container with ID starting with 835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3 not found: ID does not exist" containerID="835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3"
Nov 01 07:03:45.052078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.052020    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3"} err="failed to get container status \"835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3\": rpc error: code = NotFound desc = could not find container \"835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3\": container with ID starting with 835502e246ec646678aad65a6d46163266a00d34d517552d62954e7157222df3 not found: ID does not exist"
Nov 01 07:03:45.052078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.052040    4468 scope.go:122] "RemoveContainer" containerID="8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"
Nov 01 07:03:45.052289 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:03:45.052271    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865\": container with ID starting with 8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865 not found: ID does not exist" containerID="8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"
Nov 01 07:03:45.052333 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.052295    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865"} err="failed to get container status \"8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865\": rpc error: code = NotFound desc = could not find container \"8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865\": container with ID starting with 8b09b40a73ed1ea952be1b33280a0531dbc4757c2dc8a572489b7657f6eb3865 not found: ID does not exist"
Nov 01 07:03:45.198007 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.197973    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"]
Nov 01 07:03:45.248113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.248076    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c52bfa-openshift-default-78b4d47799-2m4jp"]
Nov 01 07:03:45.672570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.672531    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"]
Nov 01 07:03:45.673053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673040    4468 cpu_manager.go:414] "containerMap: removing container" podUID="9d3840bf-184f-401e-8fb9-52aaf400a0ec" containerName="revoke-keys"
Nov 01 07:03:45.673097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673054    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3840bf-184f-401e-8fb9-52aaf400a0ec" containerName="revoke-keys"
Nov 01 07:03:45.673097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673089    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy"
Nov 01 07:03:45.673097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673094    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy"
Nov 01 07:03:45.673188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673172    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3840bf-184f-401e-8fb9-52aaf400a0ec" containerName="revoke-keys"
Nov 01 07:03:45.673222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673189    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy"
Nov 01 07:03:45.673222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.673197    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy"
Nov 01 07:03:45.677920 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.677900    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.720414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.720202    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsg7\" (UniqueName: \"kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.720414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.720373    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.720718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.720435    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.723977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.723952    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"]
Nov 01 07:03:45.726556 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.726528    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c831a4c0-rwjh4"]
Nov 01 07:03:45.730002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.729982    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:03:45.730178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.729997    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:03:45.730178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.730001    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:03:45.730178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.730041    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:03:45.733445 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.733420    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"]
Nov 01 07:03:45.821435 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.821406    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.821637 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.821451    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.821637 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.821516    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsg7\" (UniqueName: \"kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.822210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.822187    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.823900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.823881    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.878100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.878070    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsg7\" (UniqueName: \"kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7\") pod \"maas-api-revoke-sub-695bf2a1-v2brp\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:45.991380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:45.991296    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:46.366516 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.366487    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"]
Nov 01 07:03:46.367306 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:46.367284    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2298205_fee0_4470_885b_bdb15fb85be5.slice/crio-4dcb9f5bdeb1ec3d556bf50904715f8cf33d322b0dd41887214c760d4b36bbe8 WatchSource:0}: Error finding container 4dcb9f5bdeb1ec3d556bf50904715f8cf33d322b0dd41887214c760d4b36bbe8: Status 404 returned error can't find the container with id 4dcb9f5bdeb1ec3d556bf50904715f8cf33d322b0dd41887214c760d4b36bbe8
Nov 01 07:03:46.407583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.407541    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"]
Nov 01 07:03:46.408116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.408101    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy"
Nov 01 07:03:46.408161 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.408117    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" containerName="istio-proxy"
Nov 01 07:03:46.412719 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.412699    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.427572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.427456    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.427711 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.427697    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.427755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.427718    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bkb\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.427803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.427756    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.465421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.465086    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"routing-overlay\""
Nov 01 07:03:46.465421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.465210    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25719d\""
Nov 01 07:03:46.465421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.465242    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"openshift-service-ca.crt\""
Nov 01 07:03:46.465421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.465374    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"kube-root-ca.crt\""
Nov 01 07:03:46.468867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.468845    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"]
Nov 01 07:03:46.473735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.473709    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-5e3560941b22\""
Nov 01 07:03:46.473735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.473717    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-e2e--bf6d8971-dockercfg-n4294\""
Nov 01 07:03:46.529102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.529074    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.529102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.529105    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bkb\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.529331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.529151    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.529331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.529226    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.529904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.529822    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.530013 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.529947    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.531577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.531559    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.585942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.585898    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bkb\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:46.590510 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.590458    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2a7055-d2b5-409f-aec3-6a4a9a30b17e" path="/var/lib/kubelet/pods/6b2a7055-d2b5-409f-aec3-6a4a9a30b17e/volumes"
Nov 01 07:03:46.591043 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.591022    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24aeff4-8c1e-4638-ae19-8c09010af0f6" path="/var/lib/kubelet/pods/b24aeff4-8c1e-4638-ae19-8c09010af0f6/volumes"
Nov 01 07:03:46.725364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:46.725320    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:47.048029 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:47.047994    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp" event={"ID":"c2298205-fee0-4470-885b-bdb15fb85be5","Type":"ContainerStarted","Data":"7a5c46c3b13a4191a88ae6f699b94f6cb757c3070625c8d4e74397f59886e43b"}
Nov 01 07:03:47.048204 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:47.048060    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp" event={"ID":"c2298205-fee0-4470-885b-bdb15fb85be5","Type":"ContainerStarted","Data":"4dcb9f5bdeb1ec3d556bf50904715f8cf33d322b0dd41887214c760d4b36bbe8"}
Nov 01 07:03:47.097964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:47.097933    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"]
Nov 01 07:03:47.098763 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:47.098737    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0d8558_b5ad_4991_95e8_4d4211589a15.slice/crio-f70d4251d493861e8191a0ed43a54fa63cafdf70ccd72a8e76b3e07fb8c72b9f WatchSource:0}: Error finding container f70d4251d493861e8191a0ed43a54fa63cafdf70ccd72a8e76b3e07fb8c72b9f: Status 404 returned error can't find the container with id f70d4251d493861e8191a0ed43a54fa63cafdf70ccd72a8e76b3e07fb8c72b9f
Nov 01 07:03:47.165210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:47.165160    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp" podStartSLOduration=2.165146856 podStartE2EDuration="2.165146856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:47.162497844 +0000 UTC m=+3238.730304673" watchObservedRunningTime="2026-11-01 07:03:47.165146856 +0000 UTC m=+3238.732953684"
Nov 01 07:03:47.689708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:47.689669    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:03:48.056116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:48.056076    4468 generic.go:356] "Generic (PLEG): container finished" podID="c2298205-fee0-4470-885b-bdb15fb85be5" containerID="7a5c46c3b13a4191a88ae6f699b94f6cb757c3070625c8d4e74397f59886e43b" exitCode=0
Nov 01 07:03:48.056748 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:48.056156    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp" event={"ID":"c2298205-fee0-4470-885b-bdb15fb85be5","Type":"ContainerDied","Data":"7a5c46c3b13a4191a88ae6f699b94f6cb757c3070625c8d4e74397f59886e43b"}
Nov 01 07:03:48.058049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:48.058026    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds" event={"ID":"0d0d8558-b5ad-4991-95e8-4d4211589a15","Type":"ContainerStarted","Data":"f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d"}
Nov 01 07:03:48.058157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:48.058066    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds" event={"ID":"0d0d8558-b5ad-4991-95e8-4d4211589a15","Type":"ContainerStarted","Data":"f70d4251d493861e8191a0ed43a54fa63cafdf70ccd72a8e76b3e07fb8c72b9f"}
Nov 01 07:03:48.058204 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:48.058166    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:03:48.289051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:48.288979    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds" podStartSLOduration=2.288957354 podStartE2EDuration="2.288957354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:48.287856902 +0000 UTC m=+3239.855663735" watchObservedRunningTime="2026-11-01 07:03:48.288957354 +0000 UTC m=+3239.856764183"
Nov 01 07:03:49.228203 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.228178    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:49.275811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.275775    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca\") pod \"c2298205-fee0-4470-885b-bdb15fb85be5\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") "
Nov 01 07:03:49.275979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.275876    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca\") pod \"c2298205-fee0-4470-885b-bdb15fb85be5\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") "
Nov 01 07:03:49.275979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.275930    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7\" (UniqueName: \"kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7\") pod \"c2298205-fee0-4470-885b-bdb15fb85be5\" (UID: \"c2298205-fee0-4470-885b-bdb15fb85be5\") "
Nov 01 07:03:49.276220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.276197    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca" pod "c2298205-fee0-4470-885b-bdb15fb85be5" (UID: "c2298205-fee0-4470-885b-bdb15fb85be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:49.276459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.276441    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:49.278070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.278052    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca" pod "c2298205-fee0-4470-885b-bdb15fb85be5" (UID: "c2298205-fee0-4470-885b-bdb15fb85be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:49.278278 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.278257    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7" pod "c2298205-fee0-4470-885b-bdb15fb85be5" (UID: "c2298205-fee0-4470-885b-bdb15fb85be5"). InnerVolumeSpecName "kube-api-access-ktsg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:49.377577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.377452    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2298205-fee0-4470-885b-bdb15fb85be5-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:49.377577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:49.377514    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktsg7\" (UniqueName: \"kubernetes.io/projected/c2298205-fee0-4470-885b-bdb15fb85be5-kube-api-access-ktsg7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:50.072368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:50.072337    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"
Nov 01 07:03:50.072723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:50.072338    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp" event={"ID":"c2298205-fee0-4470-885b-bdb15fb85be5","Type":"ContainerDied","Data":"4dcb9f5bdeb1ec3d556bf50904715f8cf33d322b0dd41887214c760d4b36bbe8"}
Nov 01 07:03:50.072723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:50.072504    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcb9f5bdeb1ec3d556bf50904715f8cf33d322b0dd41887214c760d4b36bbe8"
Nov 01 07:03:55.623937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.623903    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"]
Nov 01 07:03:55.624443 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.624429    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c2298205-fee0-4470-885b-bdb15fb85be5" containerName="revoke-keys"
Nov 01 07:03:55.624525 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.624444    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2298205-fee0-4470-885b-bdb15fb85be5" containerName="revoke-keys"
Nov 01 07:03:55.624597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.624586    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2298205-fee0-4470-885b-bdb15fb85be5" containerName="revoke-keys"
Nov 01 07:03:55.629626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.629609    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.684109 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.684073    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"]
Nov 01 07:03:55.729673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.729644    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c831a4c0\""
Nov 01 07:03:55.729838 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.729763    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c831a4c0-dockercfg-qv5mx\""
Nov 01 07:03:55.764993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.764944    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.765203 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.765169    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7zm\" (UniqueName: \"kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.796956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.796919    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"]
Nov 01 07:03:55.800861 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.800843    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:55.857126 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.857094    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"]
Nov 01 07:03:55.866818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.866786    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.866981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.866915    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw2x\" (UniqueName: \"kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:55.867072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.866977    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:55.867136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.867099    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7zm\" (UniqueName: \"kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.867611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.867588    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.925996 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.925923    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7zm\" (UniqueName: \"kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.942944 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.942914    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:55.968563 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.968534    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw2x\" (UniqueName: \"kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:55.968691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.968582    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:55.969327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:55.969306    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:56.025379 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:56.025346    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw2x\" (UniqueName: \"kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:56.114616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:56.114581    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:56.325727 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:56.325699    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"]
Nov 01 07:03:56.326128 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:56.326103    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda492e190_0d77_4ce3_8a81_8f3427b814be.slice/crio-aa4d94d32e1bfd9dfad8cc22e7679a053926ba58daab48e4b73082cc66337b80 WatchSource:0}: Error finding container aa4d94d32e1bfd9dfad8cc22e7679a053926ba58daab48e4b73082cc66337b80: Status 404 returned error can't find the container with id aa4d94d32e1bfd9dfad8cc22e7679a053926ba58daab48e4b73082cc66337b80
Nov 01 07:03:56.491140 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:56.491112    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"]
Nov 01 07:03:56.492253 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:03:56.492228    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bb7826_6171_4f10_96b0_0a7a063c6923.slice/crio-58bc6d5487467ef281938e8169b19208527effa9cafb7765689f5e11f45a228b WatchSource:0}: Error finding container 58bc6d5487467ef281938e8169b19208527effa9cafb7765689f5e11f45a228b: Status 404 returned error can't find the container with id 58bc6d5487467ef281938e8169b19208527effa9cafb7765689f5e11f45a228b
Nov 01 07:03:57.118673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.118633    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8" event={"ID":"a492e190-0d77-4ce3-8a81-8f3427b814be","Type":"ContainerStarted","Data":"b373b734e8ef9d4030e5d216871d4ad830f46913bb1417f43c65e8533733e243"}
Nov 01 07:03:57.119177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.118699    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8" event={"ID":"a492e190-0d77-4ce3-8a81-8f3427b814be","Type":"ContainerStarted","Data":"aa4d94d32e1bfd9dfad8cc22e7679a053926ba58daab48e4b73082cc66337b80"}
Nov 01 07:03:57.119177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.118727    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:03:57.120363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.120339    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb" event={"ID":"e9bb7826-6171-4f10-96b0-0a7a063c6923","Type":"ContainerStarted","Data":"e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3"}
Nov 01 07:03:57.120520 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.120383    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:03:57.120520 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.120399    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb" event={"ID":"e9bb7826-6171-4f10-96b0-0a7a063c6923","Type":"ContainerStarted","Data":"58bc6d5487467ef281938e8169b19208527effa9cafb7765689f5e11f45a228b"}
Nov 01 07:03:57.231413 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.231355    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8" podStartSLOduration=2.231341927 podStartE2EDuration="2.231341927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:57.230076954 +0000 UTC m=+3248.797883804" watchObservedRunningTime="2026-11-01 07:03:57.231341927 +0000 UTC m=+3248.799148755"
Nov 01 07:03:57.347290 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:57.347221    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb" podStartSLOduration=2.347201821 podStartE2EDuration="2.347201821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:03:57.341703905 +0000 UTC m=+3248.909510736" watchObservedRunningTime="2026-11-01 07:03:57.347201821 +0000 UTC m=+3248.915008651"
Nov 01 07:03:59.068750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:03:59.068723    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:04:02.947560 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:04:02.947520    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:04:08.129972 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:08.129940    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:04:08.130387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:08.130077    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:04:16.291874 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:16.291841    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:04:24.590070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:24.590040    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"]
Nov 01 07:04:24.590651 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:24.590227    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy" containerID="cri-o://0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34" gracePeriod=30
Nov 01 07:04:29.855354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.855279    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 07:04:29.981984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.981941    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982016    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982055    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982087    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982232    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:29.982353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982284    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982370    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982409    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982441    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982512    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket\") pod \"432412b2-d95a-4980-b775-494dee7a8f7c\" (UID: \"432412b2-d95a-4980-b775-494dee7a8f7c\") "
Nov 01 07:04:29.982681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982574    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:29.982681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982666    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:04:29.982863 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.982844    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:29.983145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.983013    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:29.983371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.983072    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:29.983459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.983388    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432412b2-d95a-4980-b775-494dee7a8f7c-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:29.983459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.983410    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:29.983459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.983426    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:29.984721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.984701    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:29.984795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.984753    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "kube-api-access-pmdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:29.984907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.984885    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:04:29.984907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:29.984898    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token" pod "432412b2-d95a-4980-b775-494dee7a8f7c" (UID: "432412b2-d95a-4980-b775-494dee7a8f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:30.084919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.084880    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:30.084919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.084917    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdvf\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-kube-api-access-pmdvf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:30.084919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.084927    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432412b2-d95a-4980-b775-494dee7a8f7c-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:30.085281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.084937    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432412b2-d95a-4980-b775-494dee7a8f7c-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:30.085281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.084946    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432412b2-d95a-4980-b775-494dee7a8f7c-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:30.338372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.338334    4468 generic.go:356] "Generic (PLEG): container finished" podID="432412b2-d95a-4980-b775-494dee7a8f7c" containerID="0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34" exitCode=0
Nov 01 07:04:30.338574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.338367    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" event={"ID":"432412b2-d95a-4980-b775-494dee7a8f7c","Type":"ContainerDied","Data":"0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34"}
Nov 01 07:04:30.338574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.338387    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"
Nov 01 07:04:30.338574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.338429    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn" event={"ID":"432412b2-d95a-4980-b775-494dee7a8f7c","Type":"ContainerDied","Data":"43d1734ed0859ab5b26f87b8876329e95dcb32fbbfca947e5adc09de2922c6c1"}
Nov 01 07:04:30.338574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.338455    4468 scope.go:122] "RemoveContainer" containerID="0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34"
Nov 01 07:04:30.350039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.350021    4468 scope.go:122] "RemoveContainer" containerID="ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"
Nov 01 07:04:30.360079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.360050    4468 scope.go:122] "RemoveContainer" containerID="0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34"
Nov 01 07:04:30.360449 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:04:30.360429    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34\": container with ID starting with 0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34 not found: ID does not exist" containerID="0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34"
Nov 01 07:04:30.360530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.360491    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34"} err="failed to get container status \"0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34\": rpc error: code = NotFound desc = could not find container \"0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34\": container with ID starting with 0680fb37d5d5ecf1d65f7a0a0ccaab5d6656d3596f924d85ea8b1123781c7c34 not found: ID does not exist"
Nov 01 07:04:30.360530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.360513    4468 scope.go:122] "RemoveContainer" containerID="ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"
Nov 01 07:04:30.360804 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:04:30.360787    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64\": container with ID starting with ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64 not found: ID does not exist" containerID="ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"
Nov 01 07:04:30.360854 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.360811    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64"} err="failed to get container status \"ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64\": rpc error: code = NotFound desc = could not find container \"ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64\": container with ID starting with ae47d7c27eba7aca1d4e8fdc4dda4074493a4717a2ab1d92f0b5fed7fe264f64 not found: ID does not exist"
Nov 01 07:04:30.504117 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.504084    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"]
Nov 01 07:04:30.553510 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.553453    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c79abe-openshift-default-94746d989-qknkn"]
Nov 01 07:04:30.591028 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:30.590950    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" path="/var/lib/kubelet/pods/432412b2-d95a-4980-b775-494dee7a8f7c/volumes"
Nov 01 07:04:32.948303 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:04:32.948264    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:04:44.563101 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:44.563068    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"]
Nov 01 07:04:44.563689 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:44.563314    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" containerID="cri-o://d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7" gracePeriod=30
Nov 01 07:04:47.370396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.370360    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"]
Nov 01 07:04:47.370895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.370886    4468 cpu_manager.go:414] "containerMap: removing container" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy"
Nov 01 07:04:47.370955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.370899    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy"
Nov 01 07:04:47.370955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.370915    4468 cpu_manager.go:414] "containerMap: removing container" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy"
Nov 01 07:04:47.370955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.370920    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy"
Nov 01 07:04:47.371097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.371006    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy"
Nov 01 07:04:47.371097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.371016    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="432412b2-d95a-4980-b775-494dee7a8f7c" containerName="istio-proxy"
Nov 01 07:04:47.374147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.374129    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.476544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.476514    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"e2e-conflict-a-9586208a-kserve-self-signed-certs\""
Nov 01 07:04:47.476544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.476527    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"openshift-service-ca.crt\""
Nov 01 07:04:47.476807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.476597    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"kube-root-ca.crt\""
Nov 01 07:04:47.476807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.476618    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"default-dockercfg-nhmkx\""
Nov 01 07:04:47.479592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.479569    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"]
Nov 01 07:04:47.490176 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.490149    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6xp\" (UniqueName: \"kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.490287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.490186    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.490287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.490222    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.490400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.490381    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.490457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.490441    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.490553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.490504    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592201 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592169    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592219    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592255    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592324    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6xp\" (UniqueName: \"kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592360    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592398    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592618    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592685    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.592850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.592809    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.594633 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.594608    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.594972 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.594951    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.648674 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.648606    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6xp\" (UniqueName: \"kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp\") pod \"e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:47.685948 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:47.685920    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:48.064600 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:48.064574    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"]
Nov 01 07:04:48.066020 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:04:48.065994    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89c29892_f985_4cb8_9e4d_0fff2978f270.slice/crio-28ada8f44ca0e14171607145dcfb2fcbd529d953fb69b96d5ccb1c8d565611a3 WatchSource:0}: Error finding container 28ada8f44ca0e14171607145dcfb2fcbd529d953fb69b96d5ccb1c8d565611a3: Status 404 returned error can't find the container with id 28ada8f44ca0e14171607145dcfb2fcbd529d953fb69b96d5ccb1c8d565611a3
Nov 01 07:04:48.457598 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:48.457503    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp" event={"ID":"89c29892-f985-4cb8-9e4d-0fff2978f270","Type":"ContainerStarted","Data":"5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e"}
Nov 01 07:04:48.457598 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:48.457558    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp" event={"ID":"89c29892-f985-4cb8-9e4d-0fff2978f270","Type":"ContainerStarted","Data":"28ada8f44ca0e14171607145dcfb2fcbd529d953fb69b96d5ccb1c8d565611a3"}
Nov 01 07:04:48.570791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:48.570743    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp" podStartSLOduration=1.57072904 podStartE2EDuration="1.57072904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:04:48.569045819 +0000 UTC m=+3300.136852648" watchObservedRunningTime="2026-11-01 07:04:48.57072904 +0000 UTC m=+3300.138535924"
Nov 01 07:04:48.686935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:48.686897    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:49.502221 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.502190    4468 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:04:49.502691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.502240    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:04:49.826199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.826171    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 07:04:49.918508 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918441    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918565    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918663    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918692    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:49.918788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918704    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918767    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918827    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918834    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:49.918895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918867    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.918895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918889    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.919153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.918948    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert\") pod \"646fd28f-2ae4-48ac-931f-b3f12abad15e\" (UID: \"646fd28f-2ae4-48ac-931f-b3f12abad15e\") "
Nov 01 07:04:49.919153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919003    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:49.919153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919112    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:49.919359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919331    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:04:49.919666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919496    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:49.919666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919512    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:49.919666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919523    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646fd28f-2ae4-48ac-931f-b3f12abad15e-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:49.919666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919537    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:49.919666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.919550    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:49.920978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.920957    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:49.921066 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.921058    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "kube-api-access-f9kg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:49.921549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.921527    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:04:49.921666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:49.921545    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token" pod "646fd28f-2ae4-48ac-931f-b3f12abad15e" (UID: "646fd28f-2ae4-48ac-931f-b3f12abad15e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:50.020935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.020901    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:50.020935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.020928    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:50.020935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.020939    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kg4\" (UniqueName: \"kubernetes.io/projected/646fd28f-2ae4-48ac-931f-b3f12abad15e-kube-api-access-f9kg4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:50.021157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.020948    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646fd28f-2ae4-48ac-931f-b3f12abad15e-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:50.472016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.471985    4468 generic.go:356] "Generic (PLEG): container finished" podID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerID="d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7" exitCode=0
Nov 01 07:04:50.472197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.472032    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"
Nov 01 07:04:50.472197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.472070    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" event={"ID":"646fd28f-2ae4-48ac-931f-b3f12abad15e","Type":"ContainerDied","Data":"d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7"}
Nov 01 07:04:50.472197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.472129    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5" event={"ID":"646fd28f-2ae4-48ac-931f-b3f12abad15e","Type":"ContainerDied","Data":"cc8df0020c98f0e3615c9be70944198b6795b45054a45bb3810489077b9d4010"}
Nov 01 07:04:50.472197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.472158    4468 scope.go:122] "RemoveContainer" containerID="d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7"
Nov 01 07:04:50.482663 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.482646    4468 scope.go:122] "RemoveContainer" containerID="d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"
Nov 01 07:04:50.491272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.491253    4468 scope.go:122] "RemoveContainer" containerID="d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7"
Nov 01 07:04:50.491576 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:04:50.491553    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7\": container with ID starting with d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7 not found: ID does not exist" containerID="d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7"
Nov 01 07:04:50.491636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.491584    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7"} err="failed to get container status \"d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7\": rpc error: code = NotFound desc = could not find container \"d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7\": container with ID starting with d7f348de35179fe9c2dd8a3713be42373c6dab9d557400347629fd980265e2e7 not found: ID does not exist"
Nov 01 07:04:50.491636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.491604    4468 scope.go:122] "RemoveContainer" containerID="d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"
Nov 01 07:04:50.491831 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:04:50.491812    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1\": container with ID starting with d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1 not found: ID does not exist" containerID="d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"
Nov 01 07:04:50.491891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.491840    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1"} err="failed to get container status \"d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1\": rpc error: code = NotFound desc = could not find container \"d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1\": container with ID starting with d0022180e14f08ee0178a41e40849dc63dc9dd585853a4eca57027c5fcb866e1 not found: ID does not exist"
Nov 01 07:04:50.688287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.688252    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"]
Nov 01 07:04:50.737879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:50.737798    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-vvrc5"]
Nov 01 07:04:52.589998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:52.589955    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" path="/var/lib/kubelet/pods/646fd28f-2ae4-48ac-931f-b3f12abad15e/volumes"
Nov 01 07:04:57.686330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:57.686296    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:57.698841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:57.698815    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:04:58.542968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:04:58.542933    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:05:02.947765 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:05:02.947723    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:05:03.243287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:03.243203    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:05:10.047363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:10.047309    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"]
Nov 01 07:05:10.094258 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:10.094221    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8875ea-x88jt"]
Nov 01 07:05:10.591599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:10.591566    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae" path="/var/lib/kubelet/pods/5d7452fc-4b86-416e-ac9e-2c1b29b2d1ae/volumes"
Nov 01 07:05:13.576113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.576071    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"]
Nov 01 07:05:13.576955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.576932    4468 cpu_manager.go:414] "containerMap: removing container" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy"
Nov 01 07:05:13.577015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.576960    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy"
Nov 01 07:05:13.577015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.576987    4468 cpu_manager.go:414] "containerMap: removing container" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy"
Nov 01 07:05:13.577015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.576995    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy"
Nov 01 07:05:13.577146 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.577134    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy"
Nov 01 07:05:13.582683 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.582656    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.613011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.612969    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.613186 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.613154    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fm2\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.613240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.613191    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.613240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.613208    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.635429 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.635389    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"]
Nov 01 07:05:13.681450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.681412    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"routing-overlay\""
Nov 01 07:05:13.681450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.681431    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ff1dc\""
Nov 01 07:05:13.681450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.681412    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-3e5449243430\""
Nov 01 07:05:13.681450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.681450    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"openshift-service-ca.crt\""
Nov 01 07:05:13.681764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.681531    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-e2e--528f7081-dockercfg-zq99w\""
Nov 01 07:05:13.681764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.681540    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"kube-root-ca.crt\""
Nov 01 07:05:13.714389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.714355    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.714614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.714457    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fm2\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.714614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.714496    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.714614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.714516    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.715232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.715207    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.715363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.715342    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.717087 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.717069    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.771153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.771112    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fm2\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:13.896757 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:13.896658    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:14.303822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:14.303791    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"]
Nov 01 07:05:14.305194 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:05:14.305161    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac94b8f5_03d9_4fb0_80ae_918f8c86cca8.slice/crio-38e41cdadbbb3a6aede2162fc60884f05f9a31e42844809c7dd9875bfc70b377 WatchSource:0}: Error finding container 38e41cdadbbb3a6aede2162fc60884f05f9a31e42844809c7dd9875bfc70b377: Status 404 returned error can't find the container with id 38e41cdadbbb3a6aede2162fc60884f05f9a31e42844809c7dd9875bfc70b377
Nov 01 07:05:14.632991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:14.632896    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8" event={"ID":"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8","Type":"ContainerStarted","Data":"e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8"}
Nov 01 07:05:14.632991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:14.632957    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8" event={"ID":"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8","Type":"ContainerStarted","Data":"38e41cdadbbb3a6aede2162fc60884f05f9a31e42844809c7dd9875bfc70b377"}
Nov 01 07:05:14.632991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:14.632986    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:14.746552 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:14.746483    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8" podStartSLOduration=1.746446672 podStartE2EDuration="1.746446672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:14.744650072 +0000 UTC m=+3326.312456901" watchObservedRunningTime="2026-11-01 07:05:14.746446672 +0000 UTC m=+3326.314253500"
Nov 01 07:05:21.044516 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:21.044458    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"]
Nov 01 07:05:21.093996 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:21.093955    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a010bc65-g5cq2"]
Nov 01 07:05:22.597625 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:22.597589    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae53244a-80f1-4f4b-823a-dad36025b759" path="/var/lib/kubelet/pods/ae53244a-80f1-4f4b-823a-dad36025b759/volumes"
Nov 01 07:05:25.642264 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:25.642233    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:05:27.573125 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.573085    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"]
Nov 01 07:05:27.575319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.575289    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="646fd28f-2ae4-48ac-931f-b3f12abad15e" containerName="istio-proxy"
Nov 01 07:05:27.580677 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.580646    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.634652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.634617    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ba2f3-openshift-default-dockercfg-nrd89\""
Nov 01 07:05:27.685927 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.685887    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.685971    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686056    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686093    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686131    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxlbw\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686167    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686236    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686325    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686357    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.686788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.686646    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"]
Nov 01 07:05:27.787483 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787436    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.787696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787491    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.787696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787540    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.787696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787569    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.787696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787655    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.787696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787689    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.787978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.787726    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxlbw\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.788258 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.788230    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.788350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.788326    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.788619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.788596    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.788841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.788817    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.788921 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.788906    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.789095 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.789044    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.791215 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.791016    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.791215 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.791180    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.810681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.810642    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.844232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.844154    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.844382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.844271    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxlbw\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:27.896116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:27.896076    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:28.333386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.333356    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"]
Nov 01 07:05:28.334520 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:05:28.334460    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d6dd7f_8540_4d4c_9b98_c6261e63a848.slice/crio-022a559a11be7bfc8159e76aca98a203315d101a46b43dd8cd72604f173a9c89 WatchSource:0}: Error finding container 022a559a11be7bfc8159e76aca98a203315d101a46b43dd8cd72604f173a9c89: Status 404 returned error can't find the container with id 022a559a11be7bfc8159e76aca98a203315d101a46b43dd8cd72604f173a9c89
Nov 01 07:05:28.337207 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.337176    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:28.337276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.337242    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:28.337276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.337269    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:28.722208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.722105    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" event={"ID":"55d6dd7f-8540-4d4c-9b98-c6261e63a848","Type":"ContainerStarted","Data":"55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098"}
Nov 01 07:05:28.722208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.722179    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" event={"ID":"55d6dd7f-8540-4d4c-9b98-c6261e63a848","Type":"ContainerStarted","Data":"022a559a11be7bfc8159e76aca98a203315d101a46b43dd8cd72604f173a9c89"}
Nov 01 07:05:28.839209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.839148    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podStartSLOduration=1.8391341890000001 podStartE2EDuration="1.839134189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:28.836550203 +0000 UTC m=+3340.404357033" watchObservedRunningTime="2026-11-01 07:05:28.839134189 +0000 UTC m=+3340.406941053"
Nov 01 07:05:28.896218 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:28.896178    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:29.898249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:29.898213    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 07:05:29.898665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:29.898264    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 07:05:30.143331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.143293    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"]
Nov 01 07:05:30.147866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.147844    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235221    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235309    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235350    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hww\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235409    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235440    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235458    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235740 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235554    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235740 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235581    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.235740 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.235634    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337003 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.336969    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337014    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337039    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337160    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337390 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337206    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337390 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337291    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337390 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337354    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337390 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337385    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337617 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337426    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hww\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.337871 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.337847    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.338168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.338142    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.338266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.338155    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.338266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.338226    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.338425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.338396    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.340053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.340021    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.340246 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.340225    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.356051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.356020    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"]
Nov 01 07:05:30.397319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.397283    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hww\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.399361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.399334    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.470697 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.470610    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:30.896491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.896431    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 07:05:30.896679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.896521    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 07:05:30.929271 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.929157    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"]
Nov 01 07:05:30.933681 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:05:30.933655    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05618588_1359_4222_8dc2_a7fc174b6ec1.slice/crio-5adf8e85d47cdef9e170a713d64d592477e5fed6240650e567222f1bb6bc66dd WatchSource:0}: Error finding container 5adf8e85d47cdef9e170a713d64d592477e5fed6240650e567222f1bb6bc66dd: Status 404 returned error can't find the container with id 5adf8e85d47cdef9e170a713d64d592477e5fed6240650e567222f1bb6bc66dd
Nov 01 07:05:30.936998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.936965    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:30.937080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.937034    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:30.937080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:30.937061    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:31.552534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:31.552502    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:05:31.760000 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:31.759961    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" event={"ID":"05618588-1359-4222-8dc2-a7fc174b6ec1","Type":"ContainerStarted","Data":"60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0"}
Nov 01 07:05:31.760187 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:31.760018    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" event={"ID":"05618588-1359-4222-8dc2-a7fc174b6ec1","Type":"ContainerStarted","Data":"5adf8e85d47cdef9e170a713d64d592477e5fed6240650e567222f1bb6bc66dd"}
Nov 01 07:05:31.897547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:31.897439    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 07:05:31.897547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:31.897523    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 07:05:31.979132 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:31.979074    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podStartSLOduration=1.9790600619999998 podStartE2EDuration="1.979060062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:31.97659424 +0000 UTC m=+3343.544401068" watchObservedRunningTime="2026-11-01 07:05:31.979060062 +0000 UTC m=+3343.546866891"
Nov 01 07:05:32.057832 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.057783    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"]
Nov 01 07:05:32.060332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.060303    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a2a501-vkp5r"]
Nov 01 07:05:32.471355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.471319    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:32.472484 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.472447    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 07:05:32.472561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.472508    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 07:05:32.590384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.590354    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ef186c-6f92-4602-90f7-e1e472229adf" path="/var/lib/kubelet/pods/e4ef186c-6f92-4602-90f7-e1e472229adf/volumes"
Nov 01 07:05:32.897370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.897338    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 07:05:32.897571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:32.897387    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 07:05:32.974038 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:05:32.974005    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:05:33.471094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:33.471056    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 07:05:33.471514 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:33.471116    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 07:05:33.897339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:33.897282    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 07:05:33.897546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:33.897346    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 07:05:34.472041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:34.472002    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 07:05:34.472488 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:34.472066    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 07:05:34.896366 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:34.896336    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 07:05:34.896562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:34.896407    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 07:05:35.471809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:35.471766    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 07:05:35.472009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:35.471842    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 07:05:35.901229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:35.901197    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:35.901732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:35.901705    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:35.902395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:35.902373    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:05:36.471706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:36.471669    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 07:05:36.471897 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:36.471727    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 07:05:37.476016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.475984    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:37.476511 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.476273    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:37.477041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.477025    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:05:37.664242 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.664206    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"]
Nov 01 07:05:37.668491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.668438    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.720396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.720362    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"e2e-conflict-b-9586208a-kserve-self-signed-certs\""
Nov 01 07:05:37.752185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.752099    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlgh\" (UniqueName: \"kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.752185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.752143    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.752363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.752324    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.752407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.752389    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.752654 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.752531    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.752916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.752891    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.770687 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.770656    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"]
Nov 01 07:05:37.853803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.853770    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlgh\" (UniqueName: \"kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.853993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.853809    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.853993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.853885    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.853993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.853952    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.854165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.854025    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.854220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.854169    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.854532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.854506    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.854532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.854518    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.854873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.854703    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.856438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.856411    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.856583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.856530    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.911094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.911055    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlgh\" (UniqueName: \"kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh\") pod \"e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") " pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:37.981784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:37.981743    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:38.361404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:38.361374    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"]
Nov 01 07:05:38.362415 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:05:38.362386    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a9dbc7e_9e06_4fb6_a0e5_c7d56bc00bb7.slice/crio-9b04dd899b4a7bd0aaee16d4a94093487db282474619dca09dd9613744bfeb45 WatchSource:0}: Error finding container 9b04dd899b4a7bd0aaee16d4a94093487db282474619dca09dd9613744bfeb45: Status 404 returned error can't find the container with id 9b04dd899b4a7bd0aaee16d4a94093487db282474619dca09dd9613744bfeb45
Nov 01 07:05:38.812978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:38.812939    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84" event={"ID":"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7","Type":"ContainerStarted","Data":"54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e"}
Nov 01 07:05:38.813399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:38.812991    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84" event={"ID":"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7","Type":"ContainerStarted","Data":"9b04dd899b4a7bd0aaee16d4a94093487db282474619dca09dd9613744bfeb45"}
Nov 01 07:05:38.927388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:38.927321    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84" podStartSLOduration=1.927302719 podStartE2EDuration="1.927302719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:38.925184417 +0000 UTC m=+3350.492991246" watchObservedRunningTime="2026-11-01 07:05:38.927302719 +0000 UTC m=+3350.495109550"
Nov 01 07:05:38.982221 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:38.982187    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:43.045786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:43.045752    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"]
Nov 01 07:05:43.094442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:43.094409    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c75cb7-mpqhs"]
Nov 01 07:05:44.591041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:44.591003    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9ed568-9493-404b-9a34-c9686e3e5a01" path="/var/lib/kubelet/pods/fe9ed568-9493-404b-9a34-c9686e3e5a01/volumes"
Nov 01 07:05:45.266386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.266354    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"]
Nov 01 07:05:45.271916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.271893    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350251 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350218    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350258    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350300    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350404    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350488    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350629 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350590    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350628    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6px\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350693    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.350750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.350718    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452148 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452099    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452192    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452231    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452290    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452434    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452521    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6px\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452572    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452608    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452686    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452729    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452873    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452950    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.452977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.452965    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.453093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.453063    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.454819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.454797    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.455171 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.455152    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.474069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.474035    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"]
Nov 01 07:05:45.515284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.515254    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.515493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.515325    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6px\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:45.586521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:45.586491    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:46.016122 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:46.016086    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"]
Nov 01 07:05:46.017546 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:05:46.017520    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68412ba7_1e88_4576_87b4_13872565fe67.slice/crio-2b1a203286b93989cd3802557c6a33d145db33aa39d7ef85507683e157f3aff2 WatchSource:0}: Error finding container 2b1a203286b93989cd3802557c6a33d145db33aa39d7ef85507683e157f3aff2: Status 404 returned error can't find the container with id 2b1a203286b93989cd3802557c6a33d145db33aa39d7ef85507683e157f3aff2
Nov 01 07:05:46.020077 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:46.020046    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:46.020149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:46.020111    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:46.020149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:46.020140    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:05:46.877881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:46.877845    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" event={"ID":"68412ba7-1e88-4576-87b4-13872565fe67","Type":"ContainerStarted","Data":"3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02"}
Nov 01 07:05:46.878080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:46.877908    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" event={"ID":"68412ba7-1e88-4576-87b4-13872565fe67","Type":"ContainerStarted","Data":"2b1a203286b93989cd3802557c6a33d145db33aa39d7ef85507683e157f3aff2"}
Nov 01 07:05:47.093254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:47.093201    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podStartSLOduration=2.093185443 podStartE2EDuration="2.093185443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:47.042053982 +0000 UTC m=+3358.609860835" watchObservedRunningTime="2026-11-01 07:05:47.093185443 +0000 UTC m=+3358.660992271"
Nov 01 07:05:47.587077 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:47.587043    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:47.588234 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:47.588205    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 01 07:05:47.588342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:47.588251    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 01 07:05:47.982339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:47.982257    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:47.994741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:47.994716    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:48.586919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:48.586884    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 01 07:05:48.587344 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:48.586950    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 01 07:05:48.902369 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:48.902283    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:05:49.587803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:49.587764    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 01 07:05:49.588208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:49.587827    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 01 07:05:50.587000 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:50.586970    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 01 07:05:50.587219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:50.587027    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 01 07:05:51.587804 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:51.587765    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 01 07:05:51.588230 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:51.587821    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 01 07:05:52.593406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:52.593379    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:52.593896 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:52.593807    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:05:52.594676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:05:52.594657    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:06:00.393736 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.393660    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"]
Nov 01 07:06:00.397876 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.397860    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460086 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460050    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460110    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460165    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460229    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460250    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460405 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460356    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460405 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460400    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460483 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460416    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.460545 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.460525    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652b4\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.494066 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.494010    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"]
Nov 01 07:06:00.561522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561486    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.561725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561560    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.561725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561585    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.561725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561678    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.561898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561746    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.561898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561770    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.561898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561829    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652b4\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561906    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.561989    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.562047    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562290 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.562269    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562347 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.562302    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.562551    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.562845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.562823    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.564668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.564643    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.565042 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.565022    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.619634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.619600    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.619789 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.619601    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-652b4\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:00.711945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:00.711859    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:01.140969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:01.140932    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"]
Nov 01 07:06:01.143242 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:01.143214    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd526204_e6a2_4d71_9e74_d87526b029e3.slice/crio-962ebb7dddcca1db7a43e050a175be6743481ef5898faa271b8fb9a6a9d2754d WatchSource:0}: Error finding container 962ebb7dddcca1db7a43e050a175be6743481ef5898faa271b8fb9a6a9d2754d: Status 404 returned error can't find the container with id 962ebb7dddcca1db7a43e050a175be6743481ef5898faa271b8fb9a6a9d2754d
Nov 01 07:06:01.147232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:01.147184    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:06:01.147318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:01.147285    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:06:01.147365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:01.147333    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:06:02.025489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:02.022198    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" event={"ID":"cd526204-e6a2-4d71-9e74-d87526b029e3","Type":"ContainerStarted","Data":"afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059"}
Nov 01 07:06:02.025489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:02.022288    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" event={"ID":"cd526204-e6a2-4d71-9e74-d87526b029e3","Type":"ContainerStarted","Data":"962ebb7dddcca1db7a43e050a175be6743481ef5898faa271b8fb9a6a9d2754d"}
Nov 01 07:06:02.190243 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:02.190082    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podStartSLOduration=2.190063289 podStartE2EDuration="2.190063289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:02.189418797 +0000 UTC m=+3373.757225628" watchObservedRunningTime="2026-11-01 07:06:02.190063289 +0000 UTC m=+3373.757870121"
Nov 01 07:06:02.714918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:02.714881    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:02.727856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:02.727819    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 01 07:06:02.728047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:02.727887    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 01 07:06:02.957925 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:06:02.957887    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:06:03.716188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:03.715772    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 01 07:06:03.716188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:03.715828    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 01 07:06:04.719337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:04.718885    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 01 07:06:04.719337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:04.718948    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 01 07:06:05.721492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:05.718822    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 01 07:06:05.721492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:05.718887    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 01 07:06:06.727900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:06.727870    4468 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 01 07:06:06.728350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:06.728112    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 01 07:06:07.716645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:07.716610    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:07.717057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:07.717039    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:07.717756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:07.717735    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:06:11.869170 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:11.869125    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:06:12.917983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:12.917952    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"]
Nov 01 07:06:12.923207 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:12.923186    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.022531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.022493    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"]
Nov 01 07:06:13.034483 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034423    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.034657 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034631    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.034751 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034709    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.034807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034785    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.034861 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034840    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.034949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034933    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjkx5\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.035010 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.034974    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.035072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.035009    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.035181 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.035163    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.135921 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.135888    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.135977    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136017    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136043    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136150    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136229    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjkx5\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136267    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136339 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136303    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136342    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136377    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136399    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136429    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136711    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.136900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.136858    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.138799 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.138777    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.138918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.138818    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.192875 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.192798    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.193035 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.192950    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjkx5\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.237094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.237066    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:13.660212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.660181    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"]
Nov 01 07:06:13.660604 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:13.660578    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d16f984_5a55_499f_ad7c_afc1e3d07f67.slice/crio-0e5c3ceccb34740ff15b21836f486348c8dbfe30743a6f69e47c7356a09fe07f WatchSource:0}: Error finding container 0e5c3ceccb34740ff15b21836f486348c8dbfe30743a6f69e47c7356a09fe07f: Status 404 returned error can't find the container with id 0e5c3ceccb34740ff15b21836f486348c8dbfe30743a6f69e47c7356a09fe07f
Nov 01 07:06:13.663170 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.663138    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:06:13.663252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.663201    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:06:13.663252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:13.663227    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:06:14.108478 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:14.108424    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" event={"ID":"2d16f984-5a55-499f-ad7c-afc1e3d07f67","Type":"ContainerStarted","Data":"f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67"}
Nov 01 07:06:14.108884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:14.108509    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" event={"ID":"2d16f984-5a55-499f-ad7c-afc1e3d07f67","Type":"ContainerStarted","Data":"0e5c3ceccb34740ff15b21836f486348c8dbfe30743a6f69e47c7356a09fe07f"}
Nov 01 07:06:14.244605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:14.244561    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:14.245247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:14.245207    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:14.245402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:14.245283    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:15.238211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:15.238178    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:15.238657 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:15.238241    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:15.438512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:15.438431    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podStartSLOduration=3.438416419 podStartE2EDuration="3.438416419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:14.244814463 +0000 UTC m=+3385.812621293" watchObservedRunningTime="2026-11-01 07:06:15.438416419 +0000 UTC m=+3387.006223251"
Nov 01 07:06:15.439820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:15.439802    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"]
Nov 01 07:06:16.238609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:16.238569    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:16.239121 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:16.238640    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:17.238138 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.238102    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:17.238336 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.238169    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:17.277021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.276991    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"]
Nov 01 07:06:17.281133 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.281118    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.343617 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.343588    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:06:17.347712 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.347678    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"]
Nov 01 07:06:17.383962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.383924    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:06:17.383962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.383944    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:06:17.384146 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.383924    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:06:17.391280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.391246    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzll5\" (UniqueName: \"kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.391407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.391304    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.391571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.391549    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.493555 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.493454    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.493711 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.493658    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.493855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.493839    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzll5\" (UniqueName: \"kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.494197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.494164    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.496217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.496191    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.550601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.550562    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzll5\" (UniqueName: \"kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5\") pod \"maas-api-revoke-sub-727308fb-lk8zc\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.594811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.594785    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:17.997442 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:17.997415    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7454fa9_3a84_46e0_ad1d_1a5695d46833.slice/crio-e83003b19a39e6ba2f044e7585598fae9539ae098d9c517251cbf060f8704a32 WatchSource:0}: Error finding container e83003b19a39e6ba2f044e7585598fae9539ae098d9c517251cbf060f8704a32: Status 404 returned error can't find the container with id e83003b19a39e6ba2f044e7585598fae9539ae098d9c517251cbf060f8704a32
Nov 01 07:06:17.999294 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:17.999266    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"]
Nov 01 07:06:18.153395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:18.153356    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc" event={"ID":"b7454fa9-3a84-46e0-ad1d-1a5695d46833","Type":"ContainerStarted","Data":"e83003b19a39e6ba2f044e7585598fae9539ae098d9c517251cbf060f8704a32"}
Nov 01 07:06:18.237616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:18.237588    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:18.237772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:18.237630    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:19.163075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:19.163039    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc" event={"ID":"b7454fa9-3a84-46e0-ad1d-1a5695d46833","Type":"ContainerStarted","Data":"69ce757342d7072ac3c069a6f556eb90c69b5bbd2d94b3363427d8db9e117f2f"}
Nov 01 07:06:19.237533 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:19.237498    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:19.237723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:19.237549    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:19.276060 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:19.276009    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc" podStartSLOduration=2.275997589 podStartE2EDuration="2.275997589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:19.275413778 +0000 UTC m=+3390.843220607" watchObservedRunningTime="2026-11-01 07:06:19.275997589 +0000 UTC m=+3390.843804418"
Nov 01 07:06:20.172652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:20.172614    4468 generic.go:356] "Generic (PLEG): container finished" podID="b7454fa9-3a84-46e0-ad1d-1a5695d46833" containerID="69ce757342d7072ac3c069a6f556eb90c69b5bbd2d94b3363427d8db9e117f2f" exitCode=0
Nov 01 07:06:20.173111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:20.172793    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc" event={"ID":"b7454fa9-3a84-46e0-ad1d-1a5695d46833","Type":"ContainerDied","Data":"69ce757342d7072ac3c069a6f556eb90c69b5bbd2d94b3363427d8db9e117f2f"}
Nov 01 07:06:20.237916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:20.237878    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 01 07:06:20.238070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:20.237926    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 01 07:06:21.245232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.245200    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:21.246162 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.246115    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:21.246411 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.246388    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:06:21.320795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.320768    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:21.476220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.476141    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca\") pod \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") "
Nov 01 07:06:21.476220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.476192    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca\") pod \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") "
Nov 01 07:06:21.476436 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.476330    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5\" (UniqueName: \"kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5\") pod \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\" (UID: \"b7454fa9-3a84-46e0-ad1d-1a5695d46833\") "
Nov 01 07:06:21.476591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.476568    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca" pod "b7454fa9-3a84-46e0-ad1d-1a5695d46833" (UID: "b7454fa9-3a84-46e0-ad1d-1a5695d46833"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:21.476743 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.476730    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:21.478448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.478430    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca" pod "b7454fa9-3a84-46e0-ad1d-1a5695d46833" (UID: "b7454fa9-3a84-46e0-ad1d-1a5695d46833"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:21.478448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.478438    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5" pod "b7454fa9-3a84-46e0-ad1d-1a5695d46833" (UID: "b7454fa9-3a84-46e0-ad1d-1a5695d46833"). InnerVolumeSpecName "kube-api-access-pzll5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:21.578263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.578225    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzll5\" (UniqueName: \"kubernetes.io/projected/b7454fa9-3a84-46e0-ad1d-1a5695d46833-kube-api-access-pzll5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:21.578263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.578254    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7454fa9-3a84-46e0-ad1d-1a5695d46833-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:21.765986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.765909    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"]
Nov 01 07:06:21.766491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.766451    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b7454fa9-3a84-46e0-ad1d-1a5695d46833" containerName="revoke-keys"
Nov 01 07:06:21.766491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.766479    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7454fa9-3a84-46e0-ad1d-1a5695d46833" containerName="revoke-keys"
Nov 01 07:06:21.766618 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.766608    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7454fa9-3a84-46e0-ad1d-1a5695d46833" containerName="revoke-keys"
Nov 01 07:06:21.778139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.770827    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.827036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.827002    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"]
Nov 01 07:06:21.881882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.881840    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwgd4\" (UniqueName: \"kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.882053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.881979    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.882129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.882115    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.983665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.983628    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwgd4\" (UniqueName: \"kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.983853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.983778    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.983928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.983872    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.984428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.984400    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:21.986323 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:21.986306    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:22.040038 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:22.039976    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwgd4\" (UniqueName: \"kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4\") pod \"maas-api-revoke-sub-f7217ca6-vjdpl\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:22.089794 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:22.089761    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:22.198553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:22.192529    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc" event={"ID":"b7454fa9-3a84-46e0-ad1d-1a5695d46833","Type":"ContainerDied","Data":"e83003b19a39e6ba2f044e7585598fae9539ae098d9c517251cbf060f8704a32"}
Nov 01 07:06:22.198553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:22.192592    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"
Nov 01 07:06:22.198553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:22.192599    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83003b19a39e6ba2f044e7585598fae9539ae098d9c517251cbf060f8704a32"
Nov 01 07:06:22.464827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:22.464797    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"]
Nov 01 07:06:22.465708 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:22.465685    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcca7e06_8987_4310_be1b_145ea8bc38ce.slice/crio-33b68b48816ebb4f531a9e7664a19c91b35f003c14a83dc8d904dbb5857fdf07 WatchSource:0}: Error finding container 33b68b48816ebb4f531a9e7664a19c91b35f003c14a83dc8d904dbb5857fdf07: Status 404 returned error can't find the container with id 33b68b48816ebb4f531a9e7664a19c91b35f003c14a83dc8d904dbb5857fdf07
Nov 01 07:06:23.201352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:23.201315    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl" event={"ID":"bcca7e06-8987-4310-be1b-145ea8bc38ce","Type":"ContainerStarted","Data":"e7c5a072d5b6248300893b1a06e26a6d624d9c422c1b3f0106be1e0cb6030f0e"}
Nov 01 07:06:23.201536 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:23.201377    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl" event={"ID":"bcca7e06-8987-4310-be1b-145ea8bc38ce","Type":"ContainerStarted","Data":"33b68b48816ebb4f531a9e7664a19c91b35f003c14a83dc8d904dbb5857fdf07"}
Nov 01 07:06:23.316341 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:23.316294    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl" podStartSLOduration=2.316281909 podStartE2EDuration="2.316281909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:23.313453529 +0000 UTC m=+3394.881260358" watchObservedRunningTime="2026-11-01 07:06:23.316281909 +0000 UTC m=+3394.884088737"
Nov 01 07:06:24.210492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:24.210436    4468 generic.go:356] "Generic (PLEG): container finished" podID="bcca7e06-8987-4310-be1b-145ea8bc38ce" containerID="e7c5a072d5b6248300893b1a06e26a6d624d9c422c1b3f0106be1e0cb6030f0e" exitCode=0
Nov 01 07:06:24.210492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:24.210487    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl" event={"ID":"bcca7e06-8987-4310-be1b-145ea8bc38ce","Type":"ContainerDied","Data":"e7c5a072d5b6248300893b1a06e26a6d624d9c422c1b3f0106be1e0cb6030f0e"}
Nov 01 07:06:25.354648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.354624    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:25.529023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.528946    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4\" (UniqueName: \"kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4\") pod \"bcca7e06-8987-4310-be1b-145ea8bc38ce\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") "
Nov 01 07:06:25.529184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.529087    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca\") pod \"bcca7e06-8987-4310-be1b-145ea8bc38ce\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") "
Nov 01 07:06:25.529184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.529162    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca\") pod \"bcca7e06-8987-4310-be1b-145ea8bc38ce\" (UID: \"bcca7e06-8987-4310-be1b-145ea8bc38ce\") "
Nov 01 07:06:25.529554 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.529528    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca" pod "bcca7e06-8987-4310-be1b-145ea8bc38ce" (UID: "bcca7e06-8987-4310-be1b-145ea8bc38ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:25.529678 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.529662    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:25.531698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.531670    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4" pod "bcca7e06-8987-4310-be1b-145ea8bc38ce" (UID: "bcca7e06-8987-4310-be1b-145ea8bc38ce"). InnerVolumeSpecName "kube-api-access-mwgd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:25.531800 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.531743    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca" pod "bcca7e06-8987-4310-be1b-145ea8bc38ce" (UID: "bcca7e06-8987-4310-be1b-145ea8bc38ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:25.630924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.630895    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcca7e06-8987-4310-be1b-145ea8bc38ce-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:25.630924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:25.630920    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwgd4\" (UniqueName: \"kubernetes.io/projected/bcca7e06-8987-4310-be1b-145ea8bc38ce-kube-api-access-mwgd4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:26.225090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:26.225058    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"
Nov 01 07:06:26.225281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:26.225097    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl" event={"ID":"bcca7e06-8987-4310-be1b-145ea8bc38ce","Type":"ContainerDied","Data":"33b68b48816ebb4f531a9e7664a19c91b35f003c14a83dc8d904dbb5857fdf07"}
Nov 01 07:06:26.225281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:26.225171    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b68b48816ebb4f531a9e7664a19c91b35f003c14a83dc8d904dbb5857fdf07"
Nov 01 07:06:27.028860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:27.028831    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"]
Nov 01 07:06:29.747194 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:29.747159    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"]
Nov 01 07:06:30.398256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.398222    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"]
Nov 01 07:06:30.398928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.398908    4468 cpu_manager.go:414] "containerMap: removing container" podUID="bcca7e06-8987-4310-be1b-145ea8bc38ce" containerName="revoke-keys"
Nov 01 07:06:30.398990 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.398931    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcca7e06-8987-4310-be1b-145ea8bc38ce" containerName="revoke-keys"
Nov 01 07:06:30.399039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.399030    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcca7e06-8987-4310-be1b-145ea8bc38ce" containerName="revoke-keys"
Nov 01 07:06:30.402447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.402427    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.454360 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.454326    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:06:30.454591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.454381    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:06:30.454591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.454381    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:06:30.454591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.454385    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:06:30.460655 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.460616    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd" podUID="15d82ece-9bde-4040-b7f9-aa0c891bc89c" containerName="main" containerID="cri-o://db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587" gracePeriod=45
Nov 01 07:06:30.469587 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.469557    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"]
Nov 01 07:06:30.621641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.620594    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.621641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.620727    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.621641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.620761    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djd6\" (UniqueName: \"kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.722794 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.722752    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.722979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.722858    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.722979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.722882    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djd6\" (UniqueName: \"kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.723788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.723759    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.725408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.725383    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.780025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.779994    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djd6\" (UniqueName: \"kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6\") pod \"maas-api-revoke-sub-626716ee-8jxnw\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:30.800009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.799976    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 07:06:30.823917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.823886    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir\") pod \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") "
Nov 01 07:06:30.824072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.823929    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm\") pod \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") "
Nov 01 07:06:30.824072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.823969    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache\") pod \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") "
Nov 01 07:06:30.824072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824052    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs\" (UniqueName: \"kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs\") pod \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") "
Nov 01 07:06:30.824254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824121    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7\" (UniqueName: \"kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7\") pod \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") "
Nov 01 07:06:30.824254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824179    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home\") pod \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\" (UID: \"15d82ece-9bde-4040-b7f9-aa0c891bc89c\") "
Nov 01 07:06:30.824364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824119    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir" pod "15d82ece-9bde-4040-b7f9-aa0c891bc89c" (UID: "15d82ece-9bde-4040-b7f9-aa0c891bc89c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:30.824364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824276    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache" pod "15d82ece-9bde-4040-b7f9-aa0c891bc89c" (UID: "15d82ece-9bde-4040-b7f9-aa0c891bc89c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:30.824565 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824545    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home" pod "15d82ece-9bde-4040-b7f9-aa0c891bc89c" (UID: "15d82ece-9bde-4040-b7f9-aa0c891bc89c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:30.824783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824764    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:30.824856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824787    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:30.824856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.824800    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:30.826317 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.826293    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm" pod "15d82ece-9bde-4040-b7f9-aa0c891bc89c" (UID: "15d82ece-9bde-4040-b7f9-aa0c891bc89c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:30.826513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.826490    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs" pod "15d82ece-9bde-4040-b7f9-aa0c891bc89c" (UID: "15d82ece-9bde-4040-b7f9-aa0c891bc89c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:30.826628 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.826607    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7" pod "15d82ece-9bde-4040-b7f9-aa0c891bc89c" (UID: "15d82ece-9bde-4040-b7f9-aa0c891bc89c"). InnerVolumeSpecName "kube-api-access-cvkq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:30.926293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.926258    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d82ece-9bde-4040-b7f9-aa0c891bc89c-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:30.926497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.926297    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvkq7\" (UniqueName: \"kubernetes.io/projected/15d82ece-9bde-4040-b7f9-aa0c891bc89c-kube-api-access-cvkq7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:30.926497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:30.926315    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d82ece-9bde-4040-b7f9-aa0c891bc89c-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:31.015883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.015804    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:31.260485 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.260444    4468 generic.go:356] "Generic (PLEG): container finished" podID="15d82ece-9bde-4040-b7f9-aa0c891bc89c" containerID="db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587" exitCode=0
Nov 01 07:06:31.260694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.260516    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"
Nov 01 07:06:31.260694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.260534    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd" event={"ID":"15d82ece-9bde-4040-b7f9-aa0c891bc89c","Type":"ContainerDied","Data":"db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587"}
Nov 01 07:06:31.260694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.260574    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd" event={"ID":"15d82ece-9bde-4040-b7f9-aa0c891bc89c","Type":"ContainerDied","Data":"37cac6c9ac51d2a9d4cf26ca6bfd9a8380da325e23af53a8a63a4586802aa76e"}
Nov 01 07:06:31.260694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.260590    4468 scope.go:122] "RemoveContainer" containerID="db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587"
Nov 01 07:06:31.273652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.273048    4468 scope.go:122] "RemoveContainer" containerID="db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587"
Nov 01 07:06:31.273652 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:06:31.273599    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587\": container with ID starting with db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587 not found: ID does not exist" containerID="db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587"
Nov 01 07:06:31.273759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.273638    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587"} err="failed to get container status \"db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587\": rpc error: code = NotFound desc = could not find container \"db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587\": container with ID starting with db2e78e0e7cd879cacad7299fb5b1d5c455abe9b02914df87c275c8c87d4a587 not found: ID does not exist"
Nov 01 07:06:31.396838 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.396811    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"]
Nov 01 07:06:31.397570 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:31.397544    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a002d4_d056_46d8_a899_b0a5ac168847.slice/crio-d25f46de0a0113f9c70dd071863ca8db0216d317dcb0d608158726df4c1a6ad8 WatchSource:0}: Error finding container d25f46de0a0113f9c70dd071863ca8db0216d317dcb0d608158726df4c1a6ad8: Status 404 returned error can't find the container with id d25f46de0a0113f9c70dd071863ca8db0216d317dcb0d608158726df4c1a6ad8
Nov 01 07:06:31.423643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.423612    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"]
Nov 01 07:06:31.426928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:31.426908    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c831a4c0/e2e-lifecycle-model-c831a4c0-kserve-b476c44df-785fd"]
Nov 01 07:06:32.269991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:32.269951    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw" event={"ID":"96a002d4-d056-46d8-a899-b0a5ac168847","Type":"ContainerStarted","Data":"6afe46ef81b6108fd630ba2746857511bc460f862a50317b06477eb75a054656"}
Nov 01 07:06:32.270476 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:32.270044    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw" event={"ID":"96a002d4-d056-46d8-a899-b0a5ac168847","Type":"ContainerStarted","Data":"d25f46de0a0113f9c70dd071863ca8db0216d317dcb0d608158726df4c1a6ad8"}
Nov 01 07:06:32.384474 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:32.384394    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw" podStartSLOduration=2.384375808 podStartE2EDuration="2.384375808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:32.382047871 +0000 UTC m=+3403.949854722" watchObservedRunningTime="2026-11-01 07:06:32.384375808 +0000 UTC m=+3403.952182637"
Nov 01 07:06:32.590842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:32.590811    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d82ece-9bde-4040-b7f9-aa0c891bc89c" path="/var/lib/kubelet/pods/15d82ece-9bde-4040-b7f9-aa0c891bc89c/volumes"
Nov 01 07:06:32.946842 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:06:32.946765    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:06:33.280959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:33.280870    4468 generic.go:356] "Generic (PLEG): container finished" podID="96a002d4-d056-46d8-a899-b0a5ac168847" containerID="6afe46ef81b6108fd630ba2746857511bc460f862a50317b06477eb75a054656" exitCode=0
Nov 01 07:06:33.281357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:33.280959    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw" event={"ID":"96a002d4-d056-46d8-a899-b0a5ac168847","Type":"ContainerDied","Data":"6afe46ef81b6108fd630ba2746857511bc460f862a50317b06477eb75a054656"}
Nov 01 07:06:34.446727 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.446698    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:34.469802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.469771    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6\" (UniqueName: \"kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6\") pod \"96a002d4-d056-46d8-a899-b0a5ac168847\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") "
Nov 01 07:06:34.470023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.470005    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca\") pod \"96a002d4-d056-46d8-a899-b0a5ac168847\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") "
Nov 01 07:06:34.470097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.470049    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca\") pod \"96a002d4-d056-46d8-a899-b0a5ac168847\" (UID: \"96a002d4-d056-46d8-a899-b0a5ac168847\") "
Nov 01 07:06:34.470452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.470422    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca" pod "96a002d4-d056-46d8-a899-b0a5ac168847" (UID: "96a002d4-d056-46d8-a899-b0a5ac168847"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:34.472125 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.472095    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6" pod "96a002d4-d056-46d8-a899-b0a5ac168847" (UID: "96a002d4-d056-46d8-a899-b0a5ac168847"). InnerVolumeSpecName "kube-api-access-7djd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:34.472241 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.472180    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca" pod "96a002d4-d056-46d8-a899-b0a5ac168847" (UID: "96a002d4-d056-46d8-a899-b0a5ac168847"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:34.571651 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.571620    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7djd6\" (UniqueName: \"kubernetes.io/projected/96a002d4-d056-46d8-a899-b0a5ac168847-kube-api-access-7djd6\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:34.571651 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.571649    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:34.571862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.571664    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a002d4-d056-46d8-a899-b0a5ac168847-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:34.747757 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.747709    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8" podUID="ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" containerName="payload-processing" containerID="cri-o://e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8" gracePeriod=25
Nov 01 07:06:34.764236 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.764206    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"]
Nov 01 07:06:34.764810 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.764789    4468 cpu_manager.go:414] "containerMap: removing container" podUID="96a002d4-d056-46d8-a899-b0a5ac168847" containerName="revoke-keys"
Nov 01 07:06:34.764929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.764811    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a002d4-d056-46d8-a899-b0a5ac168847" containerName="revoke-keys"
Nov 01 07:06:34.764929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.764857    4468 cpu_manager.go:414] "containerMap: removing container" podUID="15d82ece-9bde-4040-b7f9-aa0c891bc89c" containerName="main"
Nov 01 07:06:34.764929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.764865    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d82ece-9bde-4040-b7f9-aa0c891bc89c" containerName="main"
Nov 01 07:06:34.765094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.765043    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d82ece-9bde-4040-b7f9-aa0c891bc89c" containerName="main"
Nov 01 07:06:34.765094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.765071    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a002d4-d056-46d8-a899-b0a5ac168847" containerName="revoke-keys"
Nov 01 07:06:34.768351 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.768330    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.823387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.823295    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"]
Nov 01 07:06:34.875847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.875811    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.876009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.875924    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.876009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.875970    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfct\" (UniqueName: \"kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.978489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.977511    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.978489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.977595    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfct\" (UniqueName: \"kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.978489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.977833    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:34.986488 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:34.981361    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:35.026512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.023630    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:06:35.046803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.046736    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:35.054854 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.054797    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfct\" (UniqueName: \"kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct\") pod \"maas-api-revoke-sub-2ad347a4-wnl9c\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:35.084812 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.084722    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials\") pod \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") "
Nov 01 07:06:35.084997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.084848    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume\") pod \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") "
Nov 01 07:06:35.084997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.084897    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2\") pod \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") "
Nov 01 07:06:35.084997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.084928    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume\") pod \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\" (UID: \"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8\") "
Nov 01 07:06:35.085357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.085331    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume" pod "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" (UID: "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:35.085515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.085454    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume" pod "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" (UID: "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:35.085630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.085612    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:35.085683 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.085636    4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-routing-overlay-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:35.087315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.087288    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials" pod "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" (UID: "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:35.087315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.087302    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2" pod "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" (UID: "ac94b8f5-03d9-4fb0-80ae-918f8c86cca8"). InnerVolumeSpecName "kube-api-access-w9fm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:35.154873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.154839    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:35.190151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.187325    4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-provider-credentials\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:35.190151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.187378    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9fm2\" (UniqueName: \"kubernetes.io/projected/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8-kube-api-access-w9fm2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:35.296219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.296192    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"
Nov 01 07:06:35.296397 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.296234    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw" event={"ID":"96a002d4-d056-46d8-a899-b0a5ac168847","Type":"ContainerDied","Data":"d25f46de0a0113f9c70dd071863ca8db0216d317dcb0d608158726df4c1a6ad8"}
Nov 01 07:06:35.296397 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.296313    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d25f46de0a0113f9c70dd071863ca8db0216d317dcb0d608158726df4c1a6ad8"
Nov 01 07:06:35.297744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.297720    4468 generic.go:356] "Generic (PLEG): container finished" podID="ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" containerID="e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8" exitCode=0
Nov 01 07:06:35.297883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.297764    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8" event={"ID":"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8","Type":"ContainerDied","Data":"e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8"}
Nov 01 07:06:35.297883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.297777    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"
Nov 01 07:06:35.297883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.297809    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8" event={"ID":"ac94b8f5-03d9-4fb0-80ae-918f8c86cca8","Type":"ContainerDied","Data":"38e41cdadbbb3a6aede2162fc60884f05f9a31e42844809c7dd9875bfc70b377"}
Nov 01 07:06:35.297883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.297832    4468 scope.go:122] "RemoveContainer" containerID="e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8"
Nov 01 07:06:35.307857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.307841    4468 scope.go:122] "RemoveContainer" containerID="e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8"
Nov 01 07:06:35.308153 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:06:35.308137    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8\": container with ID starting with e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8 not found: ID does not exist" containerID="e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8"
Nov 01 07:06:35.308220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.308165    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8"} err="failed to get container status \"e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8\": rpc error: code = NotFound desc = could not find container \"e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8\": container with ID starting with e3a43b64af5c8eae36527feab00ab16b73210e16147b5ddf4f7c8e638d7f2cd8 not found: ID does not exist"
Nov 01 07:06:35.461706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.461647    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"]
Nov 01 07:06:35.464696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.464670    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-75bcsgtq8"]
Nov 01 07:06:35.573105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:35.573080    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"]
Nov 01 07:06:35.575817 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:35.575785    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffdae7b_aff3_4318_b8a3_a31b4cc28aed.slice/crio-c0803ef7694c715c48eee8d976273a9c98c849458aa5713dea8692000c01701f WatchSource:0}: Error finding container c0803ef7694c715c48eee8d976273a9c98c849458aa5713dea8692000c01701f: Status 404 returned error can't find the container with id c0803ef7694c715c48eee8d976273a9c98c849458aa5713dea8692000c01701f
Nov 01 07:06:36.307349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:36.307310    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c" event={"ID":"dffdae7b-aff3-4318-b8a3-a31b4cc28aed","Type":"ContainerStarted","Data":"0c30d042980f400bbd950225f4e7862ede007cf0b5d7f6db61d0f933eaacd54a"}
Nov 01 07:06:36.307571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:36.307366    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c" event={"ID":"dffdae7b-aff3-4318-b8a3-a31b4cc28aed","Type":"ContainerStarted","Data":"c0803ef7694c715c48eee8d976273a9c98c849458aa5713dea8692000c01701f"}
Nov 01 07:06:36.422380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:36.422334    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c" podStartSLOduration=2.422319998 podStartE2EDuration="2.422319998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:36.420637501 +0000 UTC m=+3407.988444327" watchObservedRunningTime="2026-11-01 07:06:36.422319998 +0000 UTC m=+3407.990126826"
Nov 01 07:06:36.590992 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:36.590924    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" path="/var/lib/kubelet/pods/ac94b8f5-03d9-4fb0-80ae-918f8c86cca8/volumes"
Nov 01 07:06:37.315727 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:37.315683    4468 generic.go:356] "Generic (PLEG): container finished" podID="dffdae7b-aff3-4318-b8a3-a31b4cc28aed" containerID="0c30d042980f400bbd950225f4e7862ede007cf0b5d7f6db61d0f933eaacd54a" exitCode=0
Nov 01 07:06:37.315887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:37.315764    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c" event={"ID":"dffdae7b-aff3-4318-b8a3-a31b4cc28aed","Type":"ContainerDied","Data":"0c30d042980f400bbd950225f4e7862ede007cf0b5d7f6db61d0f933eaacd54a"}
Nov 01 07:06:38.478347 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.478322    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:38.528052 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.528025    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca\") pod \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") "
Nov 01 07:06:38.528184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.528090    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct\" (UniqueName: \"kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct\") pod \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") "
Nov 01 07:06:38.528388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.528361    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca\") pod \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\" (UID: \"dffdae7b-aff3-4318-b8a3-a31b4cc28aed\") "
Nov 01 07:06:38.528388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.528383    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca" pod "dffdae7b-aff3-4318-b8a3-a31b4cc28aed" (UID: "dffdae7b-aff3-4318-b8a3-a31b4cc28aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:38.528879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.528860    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:38.530285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.530261    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct" pod "dffdae7b-aff3-4318-b8a3-a31b4cc28aed" (UID: "dffdae7b-aff3-4318-b8a3-a31b4cc28aed"). InnerVolumeSpecName "kube-api-access-srfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:38.530441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.530415    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca" pod "dffdae7b-aff3-4318-b8a3-a31b4cc28aed" (UID: "dffdae7b-aff3-4318-b8a3-a31b4cc28aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:38.630779 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.630747    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:38.630956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:38.630782    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srfct\" (UniqueName: \"kubernetes.io/projected/dffdae7b-aff3-4318-b8a3-a31b4cc28aed-kube-api-access-srfct\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:39.338746 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:39.338716    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"
Nov 01 07:06:39.338746 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:39.338724    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c" event={"ID":"dffdae7b-aff3-4318-b8a3-a31b4cc28aed","Type":"ContainerDied","Data":"c0803ef7694c715c48eee8d976273a9c98c849458aa5713dea8692000c01701f"}
Nov 01 07:06:39.338955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:39.338787    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0803ef7694c715c48eee8d976273a9c98c849458aa5713dea8692000c01701f"
Nov 01 07:06:42.049887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.049848    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84" podUID="8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" containerName="main" containerID="cri-o://54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e" gracePeriod=45
Nov 01 07:06:42.321634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.321608    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:06:42.359331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.359295    4468 generic.go:356] "Generic (PLEG): container finished" podID="8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" containerID="54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e" exitCode=0
Nov 01 07:06:42.359512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.359343    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"
Nov 01 07:06:42.359512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.359367    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84" event={"ID":"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7","Type":"ContainerDied","Data":"54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e"}
Nov 01 07:06:42.359512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.359396    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84" event={"ID":"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7","Type":"ContainerDied","Data":"9b04dd899b4a7bd0aaee16d4a94093487db282474619dca09dd9613744bfeb45"}
Nov 01 07:06:42.359512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.359412    4468 scope.go:122] "RemoveContainer" containerID="54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e"
Nov 01 07:06:42.369117 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.369098    4468 scope.go:122] "RemoveContainer" containerID="54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e"
Nov 01 07:06:42.369377 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:06:42.369359    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e\": container with ID starting with 54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e not found: ID does not exist" containerID="54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e"
Nov 01 07:06:42.369443 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.369385    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e"} err="failed to get container status \"54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e\": rpc error: code = NotFound desc = could not find container \"54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e\": container with ID starting with 54ab299410822f01248d0e087678dcb53df5ab9ca8ad1d13e6d5f3266d15462e not found: ID does not exist"
Nov 01 07:06:42.373423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373405    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir\") pod \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") "
Nov 01 07:06:42.373517 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373439    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache\") pod \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") "
Nov 01 07:06:42.373559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373516    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home\") pod \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") "
Nov 01 07:06:42.373559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373550    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs\") pod \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") "
Nov 01 07:06:42.373652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373585    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm\") pod \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") "
Nov 01 07:06:42.373708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373646    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir" pod "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" (UID: "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:42.373708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373683    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh\" (UniqueName: \"kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh\") pod \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\" (UID: \"8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7\") "
Nov 01 07:06:42.373820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373772    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home" pod "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" (UID: "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:42.373820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.373787    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache" pod "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" (UID: "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:42.374147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.374130    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:42.374147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.374148    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:42.374274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.374156    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:42.375967 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.375947    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs" pod "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" (UID: "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:42.376049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.376015    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm" pod "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" (UID: "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:06:42.376130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.376112    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh" pod "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" (UID: "8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7"). InnerVolumeSpecName "kube-api-access-rwlgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:42.475399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.475367    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:42.475399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.475394    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:42.475399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.475404    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwlgh\" (UniqueName: \"kubernetes.io/projected/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7-kube-api-access-rwlgh\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:42.775168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.775132    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"]
Nov 01 07:06:42.778377 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:42.778353    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-b-9586208a-kserve-849d9fd44b-w5j84"]
Nov 01 07:06:44.582554 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:44.581514    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"]
Nov 01 07:06:44.594584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:44.594539    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" path="/var/lib/kubelet/pods/8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7/volumes"
Nov 01 07:06:44.936342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:44.936268    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:06:46.544175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.544143    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"]
Nov 01 07:06:46.545425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545399    4468 cpu_manager.go:414] "containerMap: removing container" podUID="8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" containerName="main"
Nov 01 07:06:46.545425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545422    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" containerName="main"
Nov 01 07:06:46.545631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545456    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" containerName="payload-processing"
Nov 01 07:06:46.545631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545479    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" containerName="payload-processing"
Nov 01 07:06:46.545631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545494    4468 cpu_manager.go:414] "containerMap: removing container" podUID="dffdae7b-aff3-4318-b8a3-a31b4cc28aed" containerName="revoke-keys"
Nov 01 07:06:46.545631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545500    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffdae7b-aff3-4318-b8a3-a31b4cc28aed" containerName="revoke-keys"
Nov 01 07:06:46.545842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545649    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9dbc7e-9e06-4fb6-a0e5-c7d56bc00bb7" containerName="main"
Nov 01 07:06:46.545842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545670    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac94b8f5-03d9-4fb0-80ae-918f8c86cca8" containerName="payload-processing"
Nov 01 07:06:46.545842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.545678    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffdae7b-aff3-4318-b8a3-a31b4cc28aed" containerName="revoke-keys"
Nov 01 07:06:46.551631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.551611    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.611822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.611790    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"]
Nov 01 07:06:46.648691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.648663    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:06:46.648834 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.648692    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:06:46.653581 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.653547    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtnqk\" (UniqueName: \"kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.653699 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.653615    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:06:46.653699 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.653615    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:06:46.653804 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.653751    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.653891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.653868    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.755160 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.755127    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.755368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.755182    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.755607 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.755567    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtnqk\" (UniqueName: \"kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.755968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.755948    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.757820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.757800    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.811911 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.811881    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtnqk\" (UniqueName: \"kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk\") pod \"maas-api-revoke-sub-0d9d352f-hvpkj\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:46.866945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:46.866924    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:47.239378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:47.239351    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"]
Nov 01 07:06:47.240614 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:06:47.240591    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811a356d_69a1_481f_89c7_fcc7e8ae1070.slice/crio-00671a378d5eac3440e3ca4f3765b2642ecf640978c7510ae382db2e498b98ef WatchSource:0}: Error finding container 00671a378d5eac3440e3ca4f3765b2642ecf640978c7510ae382db2e498b98ef: Status 404 returned error can't find the container with id 00671a378d5eac3440e3ca4f3765b2642ecf640978c7510ae382db2e498b98ef
Nov 01 07:06:47.398639 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:47.398602    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj" event={"ID":"811a356d-69a1-481f-89c7-fcc7e8ae1070","Type":"ContainerStarted","Data":"00671a378d5eac3440e3ca4f3765b2642ecf640978c7510ae382db2e498b98ef"}
Nov 01 07:06:48.406731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:48.406698    4468 generic.go:356] "Generic (PLEG): container finished" podID="811a356d-69a1-481f-89c7-fcc7e8ae1070" containerID="deafd2e6e36644e860f1b022ca078b17421fd8c0a75274a752fb7526b5c425b3" exitCode=0
Nov 01 07:06:48.407136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:48.406780    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj" event={"ID":"811a356d-69a1-481f-89c7-fcc7e8ae1070","Type":"ContainerDied","Data":"deafd2e6e36644e860f1b022ca078b17421fd8c0a75274a752fb7526b5c425b3"}
Nov 01 07:06:49.562358 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.562335    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:49.582500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.582450    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds" podUID="0d0d8558-b5ad-4991-95e8-4d4211589a15" containerName="payload-processing" containerID="cri-o://f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d" gracePeriod=25
Nov 01 07:06:49.585252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.585230    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk\" (UniqueName: \"kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk\") pod \"811a356d-69a1-481f-89c7-fcc7e8ae1070\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") "
Nov 01 07:06:49.585335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.585295    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca\") pod \"811a356d-69a1-481f-89c7-fcc7e8ae1070\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") "
Nov 01 07:06:49.585510 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.585494    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca\") pod \"811a356d-69a1-481f-89c7-fcc7e8ae1070\" (UID: \"811a356d-69a1-481f-89c7-fcc7e8ae1070\") "
Nov 01 07:06:49.585656 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.585636    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca" pod "811a356d-69a1-481f-89c7-fcc7e8ae1070" (UID: "811a356d-69a1-481f-89c7-fcc7e8ae1070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:49.586000 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.585927    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:49.587614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.587591    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca" pod "811a356d-69a1-481f-89c7-fcc7e8ae1070" (UID: "811a356d-69a1-481f-89c7-fcc7e8ae1070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:06:49.587724 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.587679    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk" pod "811a356d-69a1-481f-89c7-fcc7e8ae1070" (UID: "811a356d-69a1-481f-89c7-fcc7e8ae1070"). InnerVolumeSpecName "kube-api-access-dtnqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:49.688087 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.688017    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811a356d-69a1-481f-89c7-fcc7e8ae1070-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:49.688087 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.688055    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtnqk\" (UniqueName: \"kubernetes.io/projected/811a356d-69a1-481f-89c7-fcc7e8ae1070-kube-api-access-dtnqk\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:49.820460 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.820429    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:06:49.889915 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.889888    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume\") pod \"0d0d8558-b5ad-4991-95e8-4d4211589a15\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") "
Nov 01 07:06:49.890088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.889930    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials\") pod \"0d0d8558-b5ad-4991-95e8-4d4211589a15\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") "
Nov 01 07:06:49.890088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.890056    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume\") pod \"0d0d8558-b5ad-4991-95e8-4d4211589a15\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") "
Nov 01 07:06:49.890208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.890119    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb\") pod \"0d0d8558-b5ad-4991-95e8-4d4211589a15\" (UID: \"0d0d8558-b5ad-4991-95e8-4d4211589a15\") "
Nov 01 07:06:49.890382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.890262    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume" pod "0d0d8558-b5ad-4991-95e8-4d4211589a15" (UID: "0d0d8558-b5ad-4991-95e8-4d4211589a15"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:49.890512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.890448    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume" pod "0d0d8558-b5ad-4991-95e8-4d4211589a15" (UID: "0d0d8558-b5ad-4991-95e8-4d4211589a15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:06:49.890621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.890604    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:49.890669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.890626    4468 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d0d8558-b5ad-4991-95e8-4d4211589a15-routing-overlay-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:49.892190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.892167    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials" pod "0d0d8558-b5ad-4991-95e8-4d4211589a15" (UID: "0d0d8558-b5ad-4991-95e8-4d4211589a15"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:49.892363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.892341    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb" pod "0d0d8558-b5ad-4991-95e8-4d4211589a15" (UID: "0d0d8558-b5ad-4991-95e8-4d4211589a15"). InnerVolumeSpecName "kube-api-access-r9bkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:06:49.992293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.992216    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9bkb\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-kube-api-access-r9bkb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:49.992293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:49.992246    4468 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d0d8558-b5ad-4991-95e8-4d4211589a15-provider-credentials\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:06:50.420838 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.420805    4468 generic.go:356] "Generic (PLEG): container finished" podID="0d0d8558-b5ad-4991-95e8-4d4211589a15" containerID="f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d" exitCode=0
Nov 01 07:06:50.421021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.420857    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"
Nov 01 07:06:50.421021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.420891    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds" event={"ID":"0d0d8558-b5ad-4991-95e8-4d4211589a15","Type":"ContainerDied","Data":"f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d"}
Nov 01 07:06:50.421021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.420948    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds" event={"ID":"0d0d8558-b5ad-4991-95e8-4d4211589a15","Type":"ContainerDied","Data":"f70d4251d493861e8191a0ed43a54fa63cafdf70ccd72a8e76b3e07fb8c72b9f"}
Nov 01 07:06:50.421021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.420975    4468 scope.go:122] "RemoveContainer" containerID="f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d"
Nov 01 07:06:50.422417 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.422367    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj" event={"ID":"811a356d-69a1-481f-89c7-fcc7e8ae1070","Type":"ContainerDied","Data":"00671a378d5eac3440e3ca4f3765b2642ecf640978c7510ae382db2e498b98ef"}
Nov 01 07:06:50.422417 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.422402    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00671a378d5eac3440e3ca4f3765b2642ecf640978c7510ae382db2e498b98ef"
Nov 01 07:06:50.422557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.422444    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"
Nov 01 07:06:50.431364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.431342    4468 scope.go:122] "RemoveContainer" containerID="f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d"
Nov 01 07:06:50.431719 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:06:50.431700    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d\": container with ID starting with f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d not found: ID does not exist" containerID="f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d"
Nov 01 07:06:50.431805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.431727    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d"} err="failed to get container status \"f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d\": rpc error: code = NotFound desc = could not find container \"f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d\": container with ID starting with f9ae5dcd7d1dcc997a9109a4e13a5d6743ccc0721387c9714d7cdf89dfff9e0d not found: ID does not exist"
Nov 01 07:06:50.636685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.636644    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"]
Nov 01 07:06:50.686411 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:50.686333    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-5cc692xds"]
Nov 01 07:06:51.967551 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:51.967518    4468 scope.go:122] "RemoveContainer" containerID="ef588fd6cfa974bd0d8fb95c57e2e21a24dc6030f16998059ef6ab2b5e1d91e8"
Nov 01 07:06:51.979169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:51.979146    4468 scope.go:122] "RemoveContainer" containerID="34544e5883394cfe271f21b5b83097776d5e9c8cc462a01269add60bef4b1153"
Nov 01 07:06:51.988438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:51.988415    4468 scope.go:122] "RemoveContainer" containerID="d382b4a953894c0539366851006c22a42bdcd60a8f28c8c0fcc590f4ef3b9bd6"
Nov 01 07:06:52.002104 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:52.002079    4468 scope.go:122] "RemoveContainer" containerID="92a605da908fcbef32a58418651349ef5a0228daf5071daa5eac0d73f445e32c"
Nov 01 07:06:52.590630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:52.590596    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0d8558-b5ad-4991-95e8-4d4211589a15" path="/var/lib/kubelet/pods/0d0d8558-b5ad-4991-95e8-4d4211589a15/volumes"
Nov 01 07:06:55.288280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:55.288245    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"]
Nov 01 07:06:55.399346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:55.399311    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"]
Nov 01 07:06:58.184426 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:06:58.184389    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"]
Nov 01 07:07:00.288706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.288661    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8" podUID="a492e190-0d77-4ce3-8a81-8f3427b814be" containerName="payload-processing" containerID="cri-o://b373b734e8ef9d4030e5d216871d4ad830f46913bb1417f43c65e8533733e243" gracePeriod=25
Nov 01 07:07:00.400274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.400219    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb" podUID="e9bb7826-6171-4f10-96b0-0a7a063c6923" containerName="payload-pre-processing" containerID="cri-o://e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3" gracePeriod=25
Nov 01 07:07:00.492640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.492601    4468 generic.go:356] "Generic (PLEG): container finished" podID="a492e190-0d77-4ce3-8a81-8f3427b814be" containerID="b373b734e8ef9d4030e5d216871d4ad830f46913bb1417f43c65e8533733e243" exitCode=0
Nov 01 07:07:00.492833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.492796    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8" event={"ID":"a492e190-0d77-4ce3-8a81-8f3427b814be","Type":"ContainerDied","Data":"b373b734e8ef9d4030e5d216871d4ad830f46913bb1417f43c65e8533733e243"}
Nov 01 07:07:00.586165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.586132    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:07:00.750357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.750318    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:07:00.776343 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.776308    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm\" (UniqueName: \"kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm\") pod \"a492e190-0d77-4ce3-8a81-8f3427b814be\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") "
Nov 01 07:07:00.776575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.776374    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume\") pod \"a492e190-0d77-4ce3-8a81-8f3427b814be\" (UID: \"a492e190-0d77-4ce3-8a81-8f3427b814be\") "
Nov 01 07:07:00.777208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.776853    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume" pod "a492e190-0d77-4ce3-8a81-8f3427b814be" (UID: "a492e190-0d77-4ce3-8a81-8f3427b814be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:00.777369 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.777338    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a492e190-0d77-4ce3-8a81-8f3427b814be-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:00.783403 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.783381    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm" pod "a492e190-0d77-4ce3-8a81-8f3427b814be" (UID: "a492e190-0d77-4ce3-8a81-8f3427b814be"). InnerVolumeSpecName "kube-api-access-ct7zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:00.878878 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.878796    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume\") pod \"e9bb7826-6171-4f10-96b0-0a7a063c6923\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") "
Nov 01 07:07:00.878878 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.878867    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x\" (UniqueName: \"kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x\") pod \"e9bb7826-6171-4f10-96b0-0a7a063c6923\" (UID: \"e9bb7826-6171-4f10-96b0-0a7a063c6923\") "
Nov 01 07:07:00.879293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.879264    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume" pod "e9bb7826-6171-4f10-96b0-0a7a063c6923" (UID: "e9bb7826-6171-4f10-96b0-0a7a063c6923"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:00.879444 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.879425    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bb7826-6171-4f10-96b0-0a7a063c6923-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:00.879576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.879451    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7zm\" (UniqueName: \"kubernetes.io/projected/a492e190-0d77-4ce3-8a81-8f3427b814be-kube-api-access-ct7zm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:00.881238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.881213    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x" pod "e9bb7826-6171-4f10-96b0-0a7a063c6923" (UID: "e9bb7826-6171-4f10-96b0-0a7a063c6923"). InnerVolumeSpecName "kube-api-access-vsw2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:00.981034 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:00.980992    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsw2x\" (UniqueName: \"kubernetes.io/projected/e9bb7826-6171-4f10-96b0-0a7a063c6923-kube-api-access-vsw2x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:01.500923 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.500891    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"
Nov 01 07:07:01.501383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.500931    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8" event={"ID":"a492e190-0d77-4ce3-8a81-8f3427b814be","Type":"ContainerDied","Data":"aa4d94d32e1bfd9dfad8cc22e7679a053926ba58daab48e4b73082cc66337b80"}
Nov 01 07:07:01.501383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.500995    4468 scope.go:122] "RemoveContainer" containerID="b373b734e8ef9d4030e5d216871d4ad830f46913bb1417f43c65e8533733e243"
Nov 01 07:07:01.502352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.502335    4468 generic.go:356] "Generic (PLEG): container finished" podID="e9bb7826-6171-4f10-96b0-0a7a063c6923" containerID="e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3" exitCode=0
Nov 01 07:07:01.502424 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.502381    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb" event={"ID":"e9bb7826-6171-4f10-96b0-0a7a063c6923","Type":"ContainerDied","Data":"e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3"}
Nov 01 07:07:01.502424 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.502390    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"
Nov 01 07:07:01.502424 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.502410    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb" event={"ID":"e9bb7826-6171-4f10-96b0-0a7a063c6923","Type":"ContainerDied","Data":"58bc6d5487467ef281938e8169b19208527effa9cafb7765689f5e11f45a228b"}
Nov 01 07:07:01.516130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.516102    4468 scope.go:122] "RemoveContainer" containerID="e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3"
Nov 01 07:07:01.526952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.526929    4468 scope.go:122] "RemoveContainer" containerID="e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3"
Nov 01 07:07:01.527270 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:01.527251    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3\": container with ID starting with e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3 not found: ID does not exist" containerID="e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3"
Nov 01 07:07:01.527322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.527281    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3"} err="failed to get container status \"e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3\": rpc error: code = NotFound desc = could not find container \"e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3\": container with ID starting with e58196b6bf12e4e5c448411c237f31649d6bffc8679461e0f67c83e5ed70cfa3 not found: ID does not exist"
Nov 01 07:07:01.663594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.663508    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"]
Nov 01 07:07:01.666989 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.666962    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-lsklb"]
Nov 01 07:07:01.776086 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.776032    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"]
Nov 01 07:07:01.779058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.779031    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-mzsr8"]
Nov 01 07:07:01.936186 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936109    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"]
Nov 01 07:07:01.936658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936645    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0d0d8558-b5ad-4991-95e8-4d4211589a15" containerName="payload-processing"
Nov 01 07:07:01.936709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936659    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0d8558-b5ad-4991-95e8-4d4211589a15" containerName="payload-processing"
Nov 01 07:07:01.936709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936691    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e9bb7826-6171-4f10-96b0-0a7a063c6923" containerName="payload-pre-processing"
Nov 01 07:07:01.936709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936696    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bb7826-6171-4f10-96b0-0a7a063c6923" containerName="payload-pre-processing"
Nov 01 07:07:01.936709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936706    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a492e190-0d77-4ce3-8a81-8f3427b814be" containerName="payload-processing"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936711    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a492e190-0d77-4ce3-8a81-8f3427b814be" containerName="payload-processing"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936719    4468 cpu_manager.go:414] "containerMap: removing container" podUID="811a356d-69a1-481f-89c7-fcc7e8ae1070" containerName="revoke-keys"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936723    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811a356d-69a1-481f-89c7-fcc7e8ae1070" containerName="revoke-keys"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936791    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bb7826-6171-4f10-96b0-0a7a063c6923" containerName="payload-pre-processing"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936800    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a492e190-0d77-4ce3-8a81-8f3427b814be" containerName="payload-processing"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936805    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="811a356d-69a1-481f-89c7-fcc7e8ae1070" containerName="revoke-keys"
Nov 01 07:07:01.936844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.936815    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0d8558-b5ad-4991-95e8-4d4211589a15" containerName="payload-processing"
Nov 01 07:07:01.941721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.941703    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:01.994186 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.994156    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:07:01.994394 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.994250    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:07:01.994394 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.994293    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:07:01.994394 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.994297    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:07:01.996372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:01.996350    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"]
Nov 01 07:07:02.097866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.097826    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.098075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.097986    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.098141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.098126    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f6q\" (UniqueName: \"kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.199315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.199212    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.199519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.199350    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.199519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.199512    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f6q\" (UniqueName: \"kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.199947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.199923    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.202081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.202057    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.255850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.255813    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f6q\" (UniqueName: \"kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q\") pod \"maas-api-revoke-sub-27248075-xmlgm\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.554797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.554763    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:02.592196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.592157    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a492e190-0d77-4ce3-8a81-8f3427b814be" path="/var/lib/kubelet/pods/a492e190-0d77-4ce3-8a81-8f3427b814be/volumes"
Nov 01 07:07:02.592726 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.592704    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bb7826-6171-4f10-96b0-0a7a063c6923" path="/var/lib/kubelet/pods/e9bb7826-6171-4f10-96b0-0a7a063c6923/volumes"
Nov 01 07:07:02.929828 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:02.929793    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"]
Nov 01 07:07:02.930728 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:07:02.930698    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ec8e21_c2dd_4697_83ea_d1e78bb2e99d.slice/crio-67d89b46c29861ad8a23adf56f8971deaea17ffbb3156bc8b8925fcb424b90d4 WatchSource:0}: Error finding container 67d89b46c29861ad8a23adf56f8971deaea17ffbb3156bc8b8925fcb424b90d4: Status 404 returned error can't find the container with id 67d89b46c29861ad8a23adf56f8971deaea17ffbb3156bc8b8925fcb424b90d4
Nov 01 07:07:02.952055 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:02.952028    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:07:03.518360 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:03.518269    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm" event={"ID":"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d","Type":"ContainerStarted","Data":"51d783c519e13d4cad3b75ad45de911b01d6a4208bbf62c8e30311a933051897"}
Nov 01 07:07:03.518360 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:03.518324    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm" event={"ID":"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d","Type":"ContainerStarted","Data":"67d89b46c29861ad8a23adf56f8971deaea17ffbb3156bc8b8925fcb424b90d4"}
Nov 01 07:07:03.633172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:03.633120    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm" podStartSLOduration=2.633106243 podStartE2EDuration="2.633106243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:03.631634377 +0000 UTC m=+3435.199441207" watchObservedRunningTime="2026-11-01 07:07:03.633106243 +0000 UTC m=+3435.200913072"
Nov 01 07:07:04.526732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:04.526698    4468 generic.go:356] "Generic (PLEG): container finished" podID="85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" containerID="51d783c519e13d4cad3b75ad45de911b01d6a4208bbf62c8e30311a933051897" exitCode=0
Nov 01 07:07:04.526940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:04.526744    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm" event={"ID":"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d","Type":"ContainerDied","Data":"51d783c519e13d4cad3b75ad45de911b01d6a4208bbf62c8e30311a933051897"}
Nov 01 07:07:05.749419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.749171    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:05.789920 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.789834    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca\") pod \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") "
Nov 01 07:07:05.790105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.790011    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q\" (UniqueName: \"kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q\") pod \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") "
Nov 01 07:07:05.790105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.790094    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca\") pod \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\" (UID: \"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d\") "
Nov 01 07:07:05.790233 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.790185    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca" pod "85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" (UID: "85ec8e21-c2dd-4697-83ea-d1e78bb2e99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:05.790576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.790551    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:05.792307 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.792279    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca" pod "85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" (UID: "85ec8e21-c2dd-4697-83ea-d1e78bb2e99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:07:05.792418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.792386    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q" pod "85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" (UID: "85ec8e21-c2dd-4697-83ea-d1e78bb2e99d"). InnerVolumeSpecName "kube-api-access-58f6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:05.891553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.891524    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:05.891553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:05.891550    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f6q\" (UniqueName: \"kubernetes.io/projected/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d-kube-api-access-58f6q\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:06.544319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:06.544290    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"
Nov 01 07:07:06.544574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:06.544331    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm" event={"ID":"85ec8e21-c2dd-4697-83ea-d1e78bb2e99d","Type":"ContainerDied","Data":"67d89b46c29861ad8a23adf56f8971deaea17ffbb3156bc8b8925fcb424b90d4"}
Nov 01 07:07:06.544574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:06.544422    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d89b46c29861ad8a23adf56f8971deaea17ffbb3156bc8b8925fcb424b90d4"
Nov 01 07:07:11.526898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.526866    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"]
Nov 01 07:07:11.527405 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.527091    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy" containerID="cri-o://3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02" gracePeriod=30
Nov 01 07:07:11.540489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.537547    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"]
Nov 01 07:07:11.540489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.537862    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy" containerID="cri-o://6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f" gracePeriod=30
Nov 01 07:07:11.562407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.562375    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"]
Nov 01 07:07:11.562628 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.562574    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy" containerID="cri-o://afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059" gracePeriod=30
Nov 01 07:07:11.575385 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.575332    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"]
Nov 01 07:07:11.575643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:11.575599    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy" containerID="cri-o://60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0" gracePeriod=30
Nov 01 07:07:13.205844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.205805    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp" podUID="89c29892-f985-4cb8-9e4d-0fff2978f270" containerName="main" containerID="cri-o://5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e" gracePeriod=45
Nov 01 07:07:13.471296 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.471229    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:07:13.592252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.592219    4468 generic.go:356] "Generic (PLEG): container finished" podID="89c29892-f985-4cb8-9e4d-0fff2978f270" containerID="5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e" exitCode=0
Nov 01 07:07:13.592447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.592267    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"
Nov 01 07:07:13.592447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.592309    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp" event={"ID":"89c29892-f985-4cb8-9e4d-0fff2978f270","Type":"ContainerDied","Data":"5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e"}
Nov 01 07:07:13.592447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.592352    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp" event={"ID":"89c29892-f985-4cb8-9e4d-0fff2978f270","Type":"ContainerDied","Data":"28ada8f44ca0e14171607145dcfb2fcbd529d953fb69b96d5ccb1c8d565611a3"}
Nov 01 07:07:13.592447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.592371    4468 scope.go:122] "RemoveContainer" containerID="5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e"
Nov 01 07:07:13.603940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.603914    4468 scope.go:122] "RemoveContainer" containerID="5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e"
Nov 01 07:07:13.604304 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:13.604279    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e\": container with ID starting with 5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e not found: ID does not exist" containerID="5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e"
Nov 01 07:07:13.604401 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.604322    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e"} err="failed to get container status \"5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e\": rpc error: code = NotFound desc = could not find container \"5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e\": container with ID starting with 5c138a7ac9823f095d1ea494d667ebf0edd029e7293db7706b94574729c9032e not found: ID does not exist"
Nov 01 07:07:13.614753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.614729    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp\" (UniqueName: \"kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp\") pod \"89c29892-f985-4cb8-9e4d-0fff2978f270\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") "
Nov 01 07:07:13.614954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.614940    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir\") pod \"89c29892-f985-4cb8-9e4d-0fff2978f270\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") "
Nov 01 07:07:13.615033 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.614967    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm\") pod \"89c29892-f985-4cb8-9e4d-0fff2978f270\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") "
Nov 01 07:07:13.615097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615064    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home\") pod \"89c29892-f985-4cb8-9e4d-0fff2978f270\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") "
Nov 01 07:07:13.615152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615139    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs\" (UniqueName: \"kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs\") pod \"89c29892-f985-4cb8-9e4d-0fff2978f270\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") "
Nov 01 07:07:13.615209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615179    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache\") pod \"89c29892-f985-4cb8-9e4d-0fff2978f270\" (UID: \"89c29892-f985-4cb8-9e4d-0fff2978f270\") "
Nov 01 07:07:13.615262 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615212    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir" pod "89c29892-f985-4cb8-9e4d-0fff2978f270" (UID: "89c29892-f985-4cb8-9e4d-0fff2978f270"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:13.615415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615329    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home" pod "89c29892-f985-4cb8-9e4d-0fff2978f270" (UID: "89c29892-f985-4cb8-9e4d-0fff2978f270"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:13.615589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615563    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache" pod "89c29892-f985-4cb8-9e4d-0fff2978f270" (UID: "89c29892-f985-4cb8-9e4d-0fff2978f270"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:13.616006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615954    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:13.616006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615982    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:13.616006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.615998    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:13.617113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.617094    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp" pod "89c29892-f985-4cb8-9e4d-0fff2978f270" (UID: "89c29892-f985-4cb8-9e4d-0fff2978f270"). InnerVolumeSpecName "kube-api-access-jw6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:13.617328 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.617308    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm" pod "89c29892-f985-4cb8-9e4d-0fff2978f270" (UID: "89c29892-f985-4cb8-9e4d-0fff2978f270"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:13.617425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.617392    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs" pod "89c29892-f985-4cb8-9e4d-0fff2978f270" (UID: "89c29892-f985-4cb8-9e4d-0fff2978f270"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:07:13.717029 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.716994    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c29892-f985-4cb8-9e4d-0fff2978f270-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:13.717029 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.717029    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6xp\" (UniqueName: \"kubernetes.io/projected/89c29892-f985-4cb8-9e4d-0fff2978f270-kube-api-access-jw6xp\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:13.717238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:13.717043    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c29892-f985-4cb8-9e4d-0fff2978f270-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:14.012023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:14.011988    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"]
Nov 01 07:07:14.015907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:14.015882    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/e2e-conflict-a-9586208a-kserve-6c556bc55b-tbjsp"]
Nov 01 07:07:14.589876 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:14.589844    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c29892-f985-4cb8-9e4d-0fff2978f270" path="/var/lib/kubelet/pods/89c29892-f985-4cb8-9e4d-0fff2978f270/volumes"
Nov 01 07:07:16.192757 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.192720    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"]
Nov 01 07:07:16.193273 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.193259    4468 cpu_manager.go:414] "containerMap: removing container" podUID="85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" containerName="revoke-keys"
Nov 01 07:07:16.193352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.193274    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" containerName="revoke-keys"
Nov 01 07:07:16.193352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.193288    4468 cpu_manager.go:414] "containerMap: removing container" podUID="89c29892-f985-4cb8-9e4d-0fff2978f270" containerName="main"
Nov 01 07:07:16.193352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.193292    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c29892-f985-4cb8-9e4d-0fff2978f270" containerName="main"
Nov 01 07:07:16.193561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.193399    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" containerName="revoke-keys"
Nov 01 07:07:16.193561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.193409    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c29892-f985-4cb8-9e4d-0fff2978f270" containerName="main"
Nov 01 07:07:16.198302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.198284    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.252942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.252907    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"]
Nov 01 07:07:16.298571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.298524    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:07:16.298571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.298553    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:07:16.298855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.298654    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:07:16.300031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.300013    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:07:16.350868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.350828    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgz8s\" (UniqueName: \"kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.351081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.350955    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.351081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.351019    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.452188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.452100    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgz8s\" (UniqueName: \"kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.452389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.452196    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.452389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.452271    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.452942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.452923    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.454710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.454688    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.509022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.508986    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgz8s\" (UniqueName: \"kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s\") pod \"maas-api-revoke-sub-4625eefa-vmp7c\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.512521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.512493    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:16.837391 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.837355    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 07:07:16.913737 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:07:16.909619    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6523b8d_3f80_4283_8e57_2d8eac4528f8.slice/crio-fedc927b6217ca722d291e4ea77a8d29c5c92b1fc2fa7b51896c807d0a170b7f WatchSource:0}: Error finding container fedc927b6217ca722d291e4ea77a8d29c5c92b1fc2fa7b51896c807d0a170b7f: Status 404 returned error can't find the container with id fedc927b6217ca722d291e4ea77a8d29c5c92b1fc2fa7b51896c807d0a170b7f
Nov 01 07:07:16.942343 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.925592    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"]
Nov 01 07:07:16.942759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.937084    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962458    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962534    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962602    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962645    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962738    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962785    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.963147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.962813    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.965539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.964954    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:16.965539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.965379    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:16.965539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.965445    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:16.970899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.965902    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.970899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.965946    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket\") pod \"44f86236-8114-49f0-bb7c-fa399fde0574\" (UID: \"44f86236-8114-49f0-bb7c-fa399fde0574\") "
Nov 01 07:07:16.970899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.970646    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:16.970899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.970861    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:16.972986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.972201    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:16.972986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.972229    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f86236-8114-49f0-bb7c-fa399fde0574-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:16.972986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.972243    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:16.985495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.978096    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:16.985495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.978950    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "kube-api-access-tvhnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:16.985495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.979168    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:07:16.985495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.981721    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token" pod "44f86236-8114-49f0-bb7c-fa399fde0574" (UID: "44f86236-8114-49f0-bb7c-fa399fde0574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:16.997496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:16.997400    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:07:17.001524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.001431    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:07:17.072905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.072863    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.072905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.072910    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.072937    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.072956    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.072990    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073026    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073067    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073101    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073150    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073180    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073231    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073266    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073384    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073440    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073497    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073558    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073591    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073613    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073644    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073674    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073703    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073728    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073783    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket\") pod \"cd526204-e6a2-4d71-9e74-d87526b029e3\" (UID: \"cd526204-e6a2-4d71-9e74-d87526b029e3\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073836    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073883    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073904    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.073934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073930    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww\") pod \"05618588-1359-4222-8dc2-a7fc174b6ec1\" (UID: \"05618588-1359-4222-8dc2-a7fc174b6ec1\") "
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.073984    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data\") pod \"68412ba7-1e88-4576-87b4-13872565fe67\" (UID: \"68412ba7-1e88-4576-87b4-13872565fe67\") "
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.074227    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.074766    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.074863    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.074882    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.074925    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.074993    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075160    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075512    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.075613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075581    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075603    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075618    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f86236-8114-49f0-bb7c-fa399fde0574-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075624    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075633    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075650    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075664    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvhnb\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-kube-api-access-tvhnb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075678    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f86236-8114-49f0-bb7c-fa399fde0574-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075707    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075720    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075736    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd526204-e6a2-4d71-9e74-d87526b029e3-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075750    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075764    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f86236-8114-49f0-bb7c-fa399fde0574-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075777    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.075795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075790    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.076528 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.075989    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:17.076528 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076029    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.076528 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076294    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.076528 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076380    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:07:17.076813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076579    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "kube-api-access-652b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:17.076813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076620    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:17.076813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076686    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.076813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.076748    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.077757 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.077729    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "kube-api-access-58hww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:17.077865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.077842    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:07:17.077975 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.077950    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "kube-api-access-7s6px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:17.078315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.078290    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:17.078442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.078409    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:17.078654 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.078636    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy" pod "05618588-1359-4222-8dc2-a7fc174b6ec1" (UID: "05618588-1359-4222-8dc2-a7fc174b6ec1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.079094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.079077    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.079213 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.079192    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy" pod "cd526204-e6a2-4d71-9e74-d87526b029e3" (UID: "cd526204-e6a2-4d71-9e74-d87526b029e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:17.079496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.079457    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo" pod "68412ba7-1e88-4576-87b4-13872565fe67" (UID: "68412ba7-1e88-4576-87b4-13872565fe67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:07:17.176749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176713    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s6px\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-kube-api-access-7s6px\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.176749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176741    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-652b4\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-kube-api-access-652b4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.176749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176751    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68412ba7-1e88-4576-87b4-13872565fe67-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176762    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68412ba7-1e88-4576-87b4-13872565fe67-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176771    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176780    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176788    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176795    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176804    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176812    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05618588-1359-4222-8dc2-a7fc174b6ec1-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176819    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68412ba7-1e88-4576-87b4-13872565fe67-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176827    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176835    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176843    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hww\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-kube-api-access-58hww\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176851    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68412ba7-1e88-4576-87b4-13872565fe67-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176859    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176867    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05618588-1359-4222-8dc2-a7fc174b6ec1-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176874    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.177085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.176882    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd526204-e6a2-4d71-9e74-d87526b029e3-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:17.639012 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.638978    4468 generic.go:356] "Generic (PLEG): container finished" podID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerID="afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059" exitCode=0
Nov 01 07:07:17.639584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.639030    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"
Nov 01 07:07:17.639584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.639060    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" event={"ID":"cd526204-e6a2-4d71-9e74-d87526b029e3","Type":"ContainerDied","Data":"afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059"}
Nov 01 07:07:17.639584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.639133    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t" event={"ID":"cd526204-e6a2-4d71-9e74-d87526b029e3","Type":"ContainerDied","Data":"962ebb7dddcca1db7a43e050a175be6743481ef5898faa271b8fb9a6a9d2754d"}
Nov 01 07:07:17.639584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.639167    4468 scope.go:122] "RemoveContainer" containerID="afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059"
Nov 01 07:07:17.640958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.640938    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c" event={"ID":"b6523b8d-3f80-4283-8e57-2d8eac4528f8","Type":"ContainerStarted","Data":"eb8c34c27c1be30bf37818022e757000c8bddd8ee0df3efff7d055c7f6816e7e"}
Nov 01 07:07:17.641042 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.640983    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c" event={"ID":"b6523b8d-3f80-4283-8e57-2d8eac4528f8","Type":"ContainerStarted","Data":"fedc927b6217ca722d291e4ea77a8d29c5c92b1fc2fa7b51896c807d0a170b7f"}
Nov 01 07:07:17.642829 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.642805    4468 generic.go:356] "Generic (PLEG): container finished" podID="44f86236-8114-49f0-bb7c-fa399fde0574" containerID="6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f" exitCode=0
Nov 01 07:07:17.642922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.642851    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"
Nov 01 07:07:17.642922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.642883    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" event={"ID":"44f86236-8114-49f0-bb7c-fa399fde0574","Type":"ContainerDied","Data":"6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f"}
Nov 01 07:07:17.643023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.642932    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5" event={"ID":"44f86236-8114-49f0-bb7c-fa399fde0574","Type":"ContainerDied","Data":"12cf2681001ac6cb784689d64b10690626578d9f7e9df59735d5bf68fa326c58"}
Nov 01 07:07:17.644404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.644382    4468 generic.go:356] "Generic (PLEG): container finished" podID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerID="60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0" exitCode=0
Nov 01 07:07:17.644512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.644423    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"
Nov 01 07:07:17.644512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.644456    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" event={"ID":"05618588-1359-4222-8dc2-a7fc174b6ec1","Type":"ContainerDied","Data":"60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0"}
Nov 01 07:07:17.644606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.644519    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g" event={"ID":"05618588-1359-4222-8dc2-a7fc174b6ec1","Type":"ContainerDied","Data":"5adf8e85d47cdef9e170a713d64d592477e5fed6240650e567222f1bb6bc66dd"}
Nov 01 07:07:17.646202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.646180    4468 generic.go:356] "Generic (PLEG): container finished" podID="68412ba7-1e88-4576-87b4-13872565fe67" containerID="3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02" exitCode=0
Nov 01 07:07:17.646322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.646219    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"
Nov 01 07:07:17.646322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.646260    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" event={"ID":"68412ba7-1e88-4576-87b4-13872565fe67","Type":"ContainerDied","Data":"3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02"}
Nov 01 07:07:17.646322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.646295    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6" event={"ID":"68412ba7-1e88-4576-87b4-13872565fe67","Type":"ContainerDied","Data":"2b1a203286b93989cd3802557c6a33d145db33aa39d7ef85507683e157f3aff2"}
Nov 01 07:07:17.653838 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.653810    4468 scope.go:122] "RemoveContainer" containerID="afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059"
Nov 01 07:07:17.654215 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:17.654192    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059\": container with ID starting with afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059 not found: ID does not exist" containerID="afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059"
Nov 01 07:07:17.654315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.654227    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059"} err="failed to get container status \"afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059\": rpc error: code = NotFound desc = could not find container \"afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059\": container with ID starting with afd285e687357aea90b15d7a046c9ff67c81cfa43cc0cb5939bb47ebd8f28059 not found: ID does not exist"
Nov 01 07:07:17.654315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.654254    4468 scope.go:122] "RemoveContainer" containerID="6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f"
Nov 01 07:07:17.665522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.665493    4468 scope.go:122] "RemoveContainer" containerID="0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"
Nov 01 07:07:17.676287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.676260    4468 scope.go:122] "RemoveContainer" containerID="6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f"
Nov 01 07:07:17.676703 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:17.676684    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f\": container with ID starting with 6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f not found: ID does not exist" containerID="6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f"
Nov 01 07:07:17.676784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.676712    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f"} err="failed to get container status \"6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f\": rpc error: code = NotFound desc = could not find container \"6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f\": container with ID starting with 6d85c72c7061a49f01dafdd26baef1680b9d0f63bd718fb28f0701d94ef4702f not found: ID does not exist"
Nov 01 07:07:17.676784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.676731    4468 scope.go:122] "RemoveContainer" containerID="0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"
Nov 01 07:07:17.677051 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:17.677022    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560\": container with ID starting with 0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560 not found: ID does not exist" containerID="0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"
Nov 01 07:07:17.677099 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.677051    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560"} err="failed to get container status \"0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560\": rpc error: code = NotFound desc = could not find container \"0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560\": container with ID starting with 0eb89ea6ed40519fde6fed1751049e5861b84d1ea5d63bd179791c09ace25560 not found: ID does not exist"
Nov 01 07:07:17.677099 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.677074    4468 scope.go:122] "RemoveContainer" containerID="60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0"
Nov 01 07:07:17.687458 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.687428    4468 scope.go:122] "RemoveContainer" containerID="60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0"
Nov 01 07:07:17.687870 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:17.687851    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0\": container with ID starting with 60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0 not found: ID does not exist" containerID="60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0"
Nov 01 07:07:17.687935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.687880    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0"} err="failed to get container status \"60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0\": rpc error: code = NotFound desc = could not find container \"60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0\": container with ID starting with 60775270ff0b7e24de1226cbc185d76881737e21ff756fae1bcc3e73f75d79f0 not found: ID does not exist"
Nov 01 07:07:17.687935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.687900    4468 scope.go:122] "RemoveContainer" containerID="3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02"
Nov 01 07:07:17.698044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.698018    4468 scope.go:122] "RemoveContainer" containerID="3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02"
Nov 01 07:07:17.698353 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:17.698326    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02\": container with ID starting with 3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02 not found: ID does not exist" containerID="3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02"
Nov 01 07:07:17.698405 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.698357    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02"} err="failed to get container status \"3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02\": rpc error: code = NotFound desc = could not find container \"3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02\": container with ID starting with 3e15bd4c33785b5899d73cbeea7d7400b751c03eb7f279343afa51acebe64e02 not found: ID does not exist"
Nov 01 07:07:17.804347 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.804324    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"]
Nov 01 07:07:17.808330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.808302    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l294g"]
Nov 01 07:07:17.917998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.917965    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"]
Nov 01 07:07:17.923791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:17.923761    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-8q7d5"]
Nov 01 07:07:18.033387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.033289    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"]
Nov 01 07:07:18.037817 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.037777    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-d6d9t"]
Nov 01 07:07:18.148686 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.148619    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c" podStartSLOduration=2.148598666 podStartE2EDuration="2.148598666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:18.145502534 +0000 UTC m=+3449.713309364" watchObservedRunningTime="2026-11-01 07:07:18.148598666 +0000 UTC m=+3449.716405494"
Nov 01 07:07:18.257647 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.257596    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"]
Nov 01 07:07:18.260576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.260553    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-p2qt6"]
Nov 01 07:07:18.485867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.485827    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"]
Nov 01 07:07:18.487089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487060    4468 cpu_manager.go:414] "containerMap: removing container" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy"
Nov 01 07:07:18.487292 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487086    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy"
Nov 01 07:07:18.487349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487313    4468 cpu_manager.go:414] "containerMap: removing container" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy"
Nov 01 07:07:18.487349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487323    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy"
Nov 01 07:07:18.487349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487343    4468 cpu_manager.go:414] "containerMap: removing container" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy"
Nov 01 07:07:18.487349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487348    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy"
Nov 01 07:07:18.487493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487369    4468 cpu_manager.go:414] "containerMap: removing container" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy"
Nov 01 07:07:18.487493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487374    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy"
Nov 01 07:07:18.487493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487380    4468 cpu_manager.go:414] "containerMap: removing container" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy"
Nov 01 07:07:18.487493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487385    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy"
Nov 01 07:07:18.487619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487560    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" containerName="istio-proxy"
Nov 01 07:07:18.487619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487571    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="68412ba7-1e88-4576-87b4-13872565fe67" containerName="istio-proxy"
Nov 01 07:07:18.487619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487580    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" containerName="istio-proxy"
Nov 01 07:07:18.487619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487589    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy"
Nov 01 07:07:18.487619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.487595    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" containerName="istio-proxy"
Nov 01 07:07:18.491089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.491057    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.539184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.539143    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"]
Nov 01 07:07:18.590887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.590855    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05618588-1359-4222-8dc2-a7fc174b6ec1" path="/var/lib/kubelet/pods/05618588-1359-4222-8dc2-a7fc174b6ec1/volumes"
Nov 01 07:07:18.591323 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.591310    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f86236-8114-49f0-bb7c-fa399fde0574" path="/var/lib/kubelet/pods/44f86236-8114-49f0-bb7c-fa399fde0574/volumes"
Nov 01 07:07:18.591752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.591738    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68412ba7-1e88-4576-87b4-13872565fe67" path="/var/lib/kubelet/pods/68412ba7-1e88-4576-87b4-13872565fe67/volumes"
Nov 01 07:07:18.592430 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.592416    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd526204-e6a2-4d71-9e74-d87526b029e3" path="/var/lib/kubelet/pods/cd526204-e6a2-4d71-9e74-d87526b029e3/volumes"
Nov 01 07:07:18.600113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.600087    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.600223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.600126    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pwp\" (UniqueName: \"kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.600321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.600300    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.656995 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.656959    4468 generic.go:356] "Generic (PLEG): container finished" podID="b6523b8d-3f80-4283-8e57-2d8eac4528f8" containerID="eb8c34c27c1be30bf37818022e757000c8bddd8ee0df3efff7d055c7f6816e7e" exitCode=0
Nov 01 07:07:18.657436 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.657047    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c" event={"ID":"b6523b8d-3f80-4283-8e57-2d8eac4528f8","Type":"ContainerDied","Data":"eb8c34c27c1be30bf37818022e757000c8bddd8ee0df3efff7d055c7f6816e7e"}
Nov 01 07:07:18.701647 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.701604    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.701820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.701753    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.701820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.701809    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pwp\" (UniqueName: \"kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.702489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.702436    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.704293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.704273    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.758984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.758897    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pwp\" (UniqueName: \"kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp\") pod \"maas-api-revoke-sub-921aa85c-k5h7b\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:18.804610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:18.804573    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:19.182763 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.182717    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"]
Nov 01 07:07:19.188400 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:07:19.188358    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235d2991_3c36_4099_bdcc_8a40338de1fc.slice/crio-c9a37e3fd42eb14308a8ba7e29a93e4b292007817ff57a36ac483ed95a530936 WatchSource:0}: Error finding container c9a37e3fd42eb14308a8ba7e29a93e4b292007817ff57a36ac483ed95a530936: Status 404 returned error can't find the container with id c9a37e3fd42eb14308a8ba7e29a93e4b292007817ff57a36ac483ed95a530936
Nov 01 07:07:19.667531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.667495    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b" event={"ID":"235d2991-3c36-4099-bdcc-8a40338de1fc","Type":"ContainerStarted","Data":"db48bd4b909ee3f7448dd0bb5b6f19bf0a3be7d68a3506d683816e71e7a46e8f"}
Nov 01 07:07:19.667975 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.667552    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b" event={"ID":"235d2991-3c36-4099-bdcc-8a40338de1fc","Type":"ContainerStarted","Data":"c9a37e3fd42eb14308a8ba7e29a93e4b292007817ff57a36ac483ed95a530936"}
Nov 01 07:07:19.780831 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.780716    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b" podStartSLOduration=1.7806981309999999 podStartE2EDuration="1.780698131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:19.779152371 +0000 UTC m=+3451.346959225" watchObservedRunningTime="2026-11-01 07:07:19.780698131 +0000 UTC m=+3451.348504962"
Nov 01 07:07:19.807433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.807404    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:19.915883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.915853    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s\" (UniqueName: \"kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s\") pod \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") "
Nov 01 07:07:19.916038 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.916020    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca\") pod \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") "
Nov 01 07:07:19.916075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.916050    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca\") pod \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\" (UID: \"b6523b8d-3f80-4283-8e57-2d8eac4528f8\") "
Nov 01 07:07:19.916427 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.916398    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca" pod "b6523b8d-3f80-4283-8e57-2d8eac4528f8" (UID: "b6523b8d-3f80-4283-8e57-2d8eac4528f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:19.916584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.916564    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:19.918646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.918614    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca" pod "b6523b8d-3f80-4283-8e57-2d8eac4528f8" (UID: "b6523b8d-3f80-4283-8e57-2d8eac4528f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:07:19.918804 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:19.918681    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s" pod "b6523b8d-3f80-4283-8e57-2d8eac4528f8" (UID: "b6523b8d-3f80-4283-8e57-2d8eac4528f8"). InnerVolumeSpecName "kube-api-access-cgz8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:20.017446 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.017403    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6523b8d-3f80-4283-8e57-2d8eac4528f8-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:20.017446 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.017441    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgz8s\" (UniqueName: \"kubernetes.io/projected/b6523b8d-3f80-4283-8e57-2d8eac4528f8-kube-api-access-cgz8s\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:20.675444 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.675409    4468 generic.go:356] "Generic (PLEG): container finished" podID="235d2991-3c36-4099-bdcc-8a40338de1fc" containerID="db48bd4b909ee3f7448dd0bb5b6f19bf0a3be7d68a3506d683816e71e7a46e8f" exitCode=0
Nov 01 07:07:20.675896 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.675501    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b" event={"ID":"235d2991-3c36-4099-bdcc-8a40338de1fc","Type":"ContainerDied","Data":"db48bd4b909ee3f7448dd0bb5b6f19bf0a3be7d68a3506d683816e71e7a46e8f"}
Nov 01 07:07:20.676989 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.676970    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c" event={"ID":"b6523b8d-3f80-4283-8e57-2d8eac4528f8","Type":"ContainerDied","Data":"fedc927b6217ca722d291e4ea77a8d29c5c92b1fc2fa7b51896c807d0a170b7f"}
Nov 01 07:07:20.677080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.677005    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fedc927b6217ca722d291e4ea77a8d29c5c92b1fc2fa7b51896c807d0a170b7f"
Nov 01 07:07:20.677080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:20.676980    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"
Nov 01 07:07:21.848788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:21.848758    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:22.041245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.041156    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp\" (UniqueName: \"kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp\") pod \"235d2991-3c36-4099-bdcc-8a40338de1fc\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") "
Nov 01 07:07:22.041245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.041235    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca\") pod \"235d2991-3c36-4099-bdcc-8a40338de1fc\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") "
Nov 01 07:07:22.041438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.041275    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca\") pod \"235d2991-3c36-4099-bdcc-8a40338de1fc\" (UID: \"235d2991-3c36-4099-bdcc-8a40338de1fc\") "
Nov 01 07:07:22.041681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.041655    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca" pod "235d2991-3c36-4099-bdcc-8a40338de1fc" (UID: "235d2991-3c36-4099-bdcc-8a40338de1fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:22.041765 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.041747    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:22.043611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.043582    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca" pod "235d2991-3c36-4099-bdcc-8a40338de1fc" (UID: "235d2991-3c36-4099-bdcc-8a40338de1fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:07:22.043727 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.043659    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp" pod "235d2991-3c36-4099-bdcc-8a40338de1fc" (UID: "235d2991-3c36-4099-bdcc-8a40338de1fc"). InnerVolumeSpecName "kube-api-access-j9pwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:22.142434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.142397    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pwp\" (UniqueName: \"kubernetes.io/projected/235d2991-3c36-4099-bdcc-8a40338de1fc-kube-api-access-j9pwp\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:22.142434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.142428    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235d2991-3c36-4099-bdcc-8a40338de1fc-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:22.699450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.699420    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"
Nov 01 07:07:22.699450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.699455    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b" event={"ID":"235d2991-3c36-4099-bdcc-8a40338de1fc","Type":"ContainerDied","Data":"c9a37e3fd42eb14308a8ba7e29a93e4b292007817ff57a36ac483ed95a530936"}
Nov 01 07:07:22.699725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:22.699522    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a37e3fd42eb14308a8ba7e29a93e4b292007817ff57a36ac483ed95a530936"
Nov 01 07:07:32.071924 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:32.071829    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"]
Nov 01 07:07:32.123249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:32.123209    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2a2a66-zbqg5"]
Nov 01 07:07:32.589991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:32.589956    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae" path="/var/lib/kubelet/pods/2b7b7ff5-da13-49fb-9e5a-dfa22ffb70ae/volumes"
Nov 01 07:07:32.948290 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:07:32.948204    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:07:35.158342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:35.158298    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:07:37.438334 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.438297    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"]
Nov 01 07:07:37.438899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.438882    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b6523b8d-3f80-4283-8e57-2d8eac4528f8" containerName="revoke-keys"
Nov 01 07:07:37.438947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.438901    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6523b8d-3f80-4283-8e57-2d8eac4528f8" containerName="revoke-keys"
Nov 01 07:07:37.438947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.438926    4468 cpu_manager.go:414] "containerMap: removing container" podUID="235d2991-3c36-4099-bdcc-8a40338de1fc" containerName="revoke-keys"
Nov 01 07:07:37.438947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.438932    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235d2991-3c36-4099-bdcc-8a40338de1fc" containerName="revoke-keys"
Nov 01 07:07:37.439044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.439035    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6523b8d-3f80-4283-8e57-2d8eac4528f8" containerName="revoke-keys"
Nov 01 07:07:37.439083 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.439049    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="235d2991-3c36-4099-bdcc-8a40338de1fc" containerName="revoke-keys"
Nov 01 07:07:37.443534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.443512    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.497121 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.497090    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:07:37.499383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.499341    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"]
Nov 01 07:07:37.536109 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.536075    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.536325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.536157    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.536325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.536244    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxp6x\" (UniqueName: \"kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.545187 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.545159    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:07:37.545187 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.545181    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:07:37.545381 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.545207    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:07:37.637668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.637636    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.637863 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.637730    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.637863 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.637763    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxp6x\" (UniqueName: \"kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.638300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.638279    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.640261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.640244    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.695304 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.695237    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxp6x\" (UniqueName: \"kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x\") pod \"maas-api-revoke-sub-f82ba365-7jvq9\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:37.756617 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:37.756585    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:38.131069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:38.131039    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"]
Nov 01 07:07:38.132206 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:07:38.132176    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8c09ec_e965_4343_9adf_833560b106fd.slice/crio-1abac2fa5e8145b16273e88432a2938d07047e8aab9e4d8d1c43736d28d83397 WatchSource:0}: Error finding container 1abac2fa5e8145b16273e88432a2938d07047e8aab9e4d8d1c43736d28d83397: Status 404 returned error can't find the container with id 1abac2fa5e8145b16273e88432a2938d07047e8aab9e4d8d1c43736d28d83397
Nov 01 07:07:38.808816 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:38.808782    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9" event={"ID":"6b8c09ec-e965-4343-9adf-833560b106fd","Type":"ContainerStarted","Data":"af3a8635e5f185eeee5c0a4e28cf742c7df32e938abd3dfeed043d64c19f68da"}
Nov 01 07:07:38.809238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:38.808836    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9" event={"ID":"6b8c09ec-e965-4343-9adf-833560b106fd","Type":"ContainerStarted","Data":"1abac2fa5e8145b16273e88432a2938d07047e8aab9e4d8d1c43736d28d83397"}
Nov 01 07:07:38.924869 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:38.924813    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9" podStartSLOduration=1.9247943699999999 podStartE2EDuration="1.92479437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:38.921701158 +0000 UTC m=+3470.489507988" watchObservedRunningTime="2026-11-01 07:07:38.92479437 +0000 UTC m=+3470.492601198"
Nov 01 07:07:39.817739 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:39.817700    4468 generic.go:356] "Generic (PLEG): container finished" podID="6b8c09ec-e965-4343-9adf-833560b106fd" containerID="af3a8635e5f185eeee5c0a4e28cf742c7df32e938abd3dfeed043d64c19f68da" exitCode=0
Nov 01 07:07:39.818185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:39.817786    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9" event={"ID":"6b8c09ec-e965-4343-9adf-833560b106fd","Type":"ContainerDied","Data":"af3a8635e5f185eeee5c0a4e28cf742c7df32e938abd3dfeed043d64c19f68da"}
Nov 01 07:07:40.974526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:40.974499    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:41.083266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.083167    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x\" (UniqueName: \"kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x\") pod \"6b8c09ec-e965-4343-9adf-833560b106fd\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") "
Nov 01 07:07:41.083455 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.083374    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca\") pod \"6b8c09ec-e965-4343-9adf-833560b106fd\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") "
Nov 01 07:07:41.083455 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.083411    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca\") pod \"6b8c09ec-e965-4343-9adf-833560b106fd\" (UID: \"6b8c09ec-e965-4343-9adf-833560b106fd\") "
Nov 01 07:07:41.083766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.083745    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca" pod "6b8c09ec-e965-4343-9adf-833560b106fd" (UID: "6b8c09ec-e965-4343-9adf-833560b106fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:41.083912 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.083897    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:41.085542 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.085520    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca" pod "6b8c09ec-e965-4343-9adf-833560b106fd" (UID: "6b8c09ec-e965-4343-9adf-833560b106fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:07:41.085596 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.085569    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x" pod "6b8c09ec-e965-4343-9adf-833560b106fd" (UID: "6b8c09ec-e965-4343-9adf-833560b106fd"). InnerVolumeSpecName "kube-api-access-xxp6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:41.185232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.185192    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8c09ec-e965-4343-9adf-833560b106fd-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:41.185232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.185231    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxp6x\" (UniqueName: \"kubernetes.io/projected/6b8c09ec-e965-4343-9adf-833560b106fd-kube-api-access-xxp6x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:41.833530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.833498    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"
Nov 01 07:07:41.833530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.833510    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9" event={"ID":"6b8c09ec-e965-4343-9adf-833560b106fd","Type":"ContainerDied","Data":"1abac2fa5e8145b16273e88432a2938d07047e8aab9e4d8d1c43736d28d83397"}
Nov 01 07:07:41.833754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:41.833555    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1abac2fa5e8145b16273e88432a2938d07047e8aab9e4d8d1c43736d28d83397"
Nov 01 07:07:52.061349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:52.061308    4468 scope.go:122] "RemoveContainer" containerID="a70c82675d783e6bea5fb2f3dc1251fc8f46ff9fb2f62c1d03bc42dbd27a69d4"
Nov 01 07:07:52.773823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:52.773790    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"]
Nov 01 07:07:52.774070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:52.774026    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy" containerID="cri-o://c041c49ff9f6c0b85c32b51e61380d812b8500d3792b41647531e61b6d5682bc" gracePeriod=30
Nov 01 07:07:57.956501 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:57.956454    4468 generic.go:356] "Generic (PLEG): container finished" podID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerID="c041c49ff9f6c0b85c32b51e61380d812b8500d3792b41647531e61b6d5682bc" exitCode=0
Nov 01 07:07:57.956956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:57.956510    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" event={"ID":"1f695329-a1fa-49f4-83bc-0b64fca90d7d","Type":"ContainerDied","Data":"c041c49ff9f6c0b85c32b51e61380d812b8500d3792b41647531e61b6d5682bc"}
Nov 01 07:07:57.956956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:57.956584    4468 scope.go:122] "RemoveContainer" containerID="9c49963b9ac9318f0e7801264fd172706e8c232d9006f7e84a865645e01c1273"
Nov 01 07:07:58.045782 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.045757    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 07:07:58.086217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086181    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086217    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086255    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086301    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086369    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086391    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086412    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086454    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086557    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:58.086867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086577    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:58.086867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086629    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy\") pod \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\" (UID: \"1f695329-a1fa-49f4-83bc-0b64fca90d7d\") "
Nov 01 07:07:58.086867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086659    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:58.086867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086752    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:58.087150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.086865    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:07:58.087150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.087059    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.087150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.087079    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.087150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.087092    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.087150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.087121    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.087150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.087130    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.088904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.088876    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:07:58.088904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.088896    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:07:58.089295 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.089270    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:58.089295 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.089283    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b" pod "1f695329-a1fa-49f4-83bc-0b64fca90d7d" (UID: "1f695329-a1fa-49f4-83bc-0b64fca90d7d"). InnerVolumeSpecName "kube-api-access-kns7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:07:58.188673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.188634    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.188844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.188678    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kns7b\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-kube-api-access-kns7b\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.188844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.188695    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.188844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.188708    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f695329-a1fa-49f4-83bc-0b64fca90d7d-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:07:58.964754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.964666    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk" event={"ID":"1f695329-a1fa-49f4-83bc-0b64fca90d7d","Type":"ContainerDied","Data":"910c48ebef881453d7c257386fb43a8cbc2cda7a20c06e826e5d89a9946fd282"}
Nov 01 07:07:58.964754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.964702    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"
Nov 01 07:07:58.964754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:58.964732    4468 scope.go:122] "RemoveContainer" containerID="c041c49ff9f6c0b85c32b51e61380d812b8500d3792b41647531e61b6d5682bc"
Nov 01 07:07:59.179275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:59.179241    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"]
Nov 01 07:07:59.228291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:07:59.228210    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-7zxhk"]
Nov 01 07:08:00.619395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:00.619349    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" path="/var/lib/kubelet/pods/1f695329-a1fa-49f4-83bc-0b64fca90d7d/volumes"
Nov 01 07:08:01.661860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.661829    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"]
Nov 01 07:08:01.662345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662330    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy"
Nov 01 07:08:01.662393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662348    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy"
Nov 01 07:08:01.662393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662358    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy"
Nov 01 07:08:01.662393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662364    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy"
Nov 01 07:08:01.662393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662374    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6b8c09ec-e965-4343-9adf-833560b106fd" containerName="revoke-keys"
Nov 01 07:08:01.662393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662380    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8c09ec-e965-4343-9adf-833560b106fd" containerName="revoke-keys"
Nov 01 07:08:01.662594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662477    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy"
Nov 01 07:08:01.662594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662486    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f695329-a1fa-49f4-83bc-0b64fca90d7d" containerName="istio-proxy"
Nov 01 07:08:01.662594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.662493    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8c09ec-e965-4343-9adf-833560b106fd" containerName="revoke-keys"
Nov 01 07:08:01.667347 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.667324    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.711309 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.711265    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"]
Nov 01 07:08:01.715326 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.715304    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.720713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.720676    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g6wqc\""
Nov 01 07:08:01.730878 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.730849    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.730912    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731001    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731149    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731283    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731348 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731334    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731394    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731458    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.731539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.731521    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7992\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.816509 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.812540    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"]
Nov 01 07:08:01.816509 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.816113    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"]
Nov 01 07:08:01.832676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.832636    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.832888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.832718    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9dl\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.832888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.832756    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.832888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.832862    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.832891    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.832997    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833040    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833065    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7992\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833117    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833200    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833221    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833238    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833130    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833286    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833322    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833357    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833382    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833523    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833544    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833612    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833645    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.833882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833648    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.833882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.833749    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.835848 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.835824    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.835999 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.835965    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.889665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.889623    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.889827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.889701    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7992\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992\") pod \"maas-default-gateway-openshift-default-7d9897b54c-sjcrx\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.935256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935156    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935211    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935240    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935261    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935279    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935309    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935406    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9dl\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935491    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935526    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935652    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935710    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.935942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935767    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.936117 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.935969    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.936303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.936280    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.938244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.938223    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.938373 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.938346    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.982115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.982086    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:01.992423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.992395    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:01.992658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:01.992630    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9dl\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xqsk9\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:02.031016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.030973    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:02.410913 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.410873    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"]
Nov 01 07:08:02.453639 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:02.453607    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35610e79_3429_44ff_bda1_db811e84b394.slice/crio-da8831744d20c106607d0eea50afb4ca51d27d19eb1b53a67e3bbbac7202fcf6 WatchSource:0}: Error finding container da8831744d20c106607d0eea50afb4ca51d27d19eb1b53a67e3bbbac7202fcf6: Status 404 returned error can't find the container with id da8831744d20c106607d0eea50afb4ca51d27d19eb1b53a67e3bbbac7202fcf6
Nov 01 07:08:02.456028 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.455993    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:02.456107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.456069    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:02.456107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.456099    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:02.464621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.464595    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"]
Nov 01 07:08:02.466346 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:02.466318    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c69c45_e08c_46c5_a024_c1bd27b60f95.slice/crio-7eccd5d95f8989abe5d1a285aa40ac7ba58d015222dbb25025b7e514c5a52431 WatchSource:0}: Error finding container 7eccd5d95f8989abe5d1a285aa40ac7ba58d015222dbb25025b7e514c5a52431: Status 404 returned error can't find the container with id 7eccd5d95f8989abe5d1a285aa40ac7ba58d015222dbb25025b7e514c5a52431
Nov 01 07:08:02.468910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.468878    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:02.469001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.468948    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:02.469001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:02.468978    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:02.957162 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:02.957119    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:08:03.004049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.003686    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" event={"ID":"86c69c45-e08c-46c5-a024-c1bd27b60f95","Type":"ContainerStarted","Data":"074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299"}
Nov 01 07:08:03.004049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.003748    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" event={"ID":"86c69c45-e08c-46c5-a024-c1bd27b60f95","Type":"ContainerStarted","Data":"7eccd5d95f8989abe5d1a285aa40ac7ba58d015222dbb25025b7e514c5a52431"}
Nov 01 07:08:03.006667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.006637    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" event={"ID":"35610e79-3429-44ff-bda1-db811e84b394","Type":"ContainerStarted","Data":"5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1"}
Nov 01 07:08:03.006901 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.006879    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" event={"ID":"35610e79-3429-44ff-bda1-db811e84b394","Type":"ContainerStarted","Data":"da8831744d20c106607d0eea50afb4ca51d27d19eb1b53a67e3bbbac7202fcf6"}
Nov 01 07:08:03.040721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.040681    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:03.043074 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.043042    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:03.043329 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.043301    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:03.259819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.259614    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podStartSLOduration=2.259597737 podStartE2EDuration="2.259597737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:03.258537484 +0000 UTC m=+3494.826344327" watchObservedRunningTime="2026-11-01 07:08:03.259597737 +0000 UTC m=+3494.827404582"
Nov 01 07:08:03.260035 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.260005    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podStartSLOduration=2.259997643 podStartE2EDuration="2.259997643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:03.145210012 +0000 UTC m=+3494.713016843" watchObservedRunningTime="2026-11-01 07:08:03.259997643 +0000 UTC m=+3494.827804474"
Nov 01 07:08:03.982530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.982493    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:03.984041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.984012    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:03.984266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:03.984240    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:04.043493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:04.042542    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:04.043493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:04.042606    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:04.982679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:04.982645    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:04.983169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:04.983143    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:05.042496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:05.038717    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:05.042496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:05.038780    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:05.983485 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:05.983439    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:05.983952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:05.983524    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:06.036494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:06.035924    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:06.036494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:06.035987    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:06.982883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:06.982841    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:06.983093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:06.982906    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:07.045495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:07.044608    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:07.045495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:07.044688    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:07.983481 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:07.983435    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:07.983683 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:07.983534    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:08.035499 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:08.033595    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:08.035499 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:08.033664    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:08.983128 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:08.983089    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:08.983572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:08.983160    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:09.032105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:09.032061    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:09.032294 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:09.032130    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:09.983110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:09.983064    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-sjcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 01 07:08:09.983331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:09.983141    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 01 07:08:10.032380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:10.032343    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xqsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 01 07:08:10.032572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:10.032406    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 01 07:08:11.069417 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:11.069378    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:11.070341 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:11.070313    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:11.071144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:11.071124    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:08:11.083550 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:11.083516    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:11.083966 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:11.083910    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:11.084971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:11.084946    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:08:12.750421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:12.750378    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"]
Nov 01 07:08:12.750887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:12.750665    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy" containerID="cri-o://75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a" gracePeriod=30
Nov 01 07:08:15.281962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.281927    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"]
Nov 01 07:08:15.286179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.286157    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.338777 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.338723    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:08:15.354510 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.354437    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9md7\" (UniqueName: \"kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.354698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.354651    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.354752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.354704    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.388440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.388401    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:08:15.388662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.388505    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:08:15.388662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.388537    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:08:15.391849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.391825    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"]
Nov 01 07:08:15.456448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.456406    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.456448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.456453    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.456763 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.456710    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9md7\" (UniqueName: \"kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.457100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.457078    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.459197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.459171    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.513233 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.513194    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9md7\" (UniqueName: \"kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7\") pod \"maas-api-revoke-sub-a4206727-jppvd\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.600730 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.600688    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:15.981243 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:15.981208    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"]
Nov 01 07:08:15.982440 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:15.982411    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bf930b_43d5_4357_8ff2_826c5940b15f.slice/crio-d7037183af1b24a238f083ffcd9b958ba01dd5abaf83e5e89520867425843524 WatchSource:0}: Error finding container d7037183af1b24a238f083ffcd9b958ba01dd5abaf83e5e89520867425843524: Status 404 returned error can't find the container with id d7037183af1b24a238f083ffcd9b958ba01dd5abaf83e5e89520867425843524
Nov 01 07:08:16.140234 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.140194    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd" event={"ID":"43bf930b-43d5-4357-8ff2-826c5940b15f","Type":"ContainerStarted","Data":"d7037183af1b24a238f083ffcd9b958ba01dd5abaf83e5e89520867425843524"}
Nov 01 07:08:16.790351 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.790266    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"]
Nov 01 07:08:16.794393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.794365    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.840331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.840284    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"]
Nov 01 07:08:16.844391 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.844370    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.874385 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874346    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874395    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874561    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874592    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874690    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874730    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874759    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874854    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qxz\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.874993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.874920    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.890710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.890669    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"]
Nov 01 07:08:16.901321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.901278    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"]
Nov 01 07:08:16.975823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.975786    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.975823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.975830    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.975901    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qxz\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.975930    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.975950    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.975989    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976019    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976124    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976175    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976212    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976288    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976300    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976327    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976359    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlgn\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976393    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976498    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976568    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976591    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976605    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.976723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976705    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:16.977282 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.976782    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.977282 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.977052    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.977282 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.977219    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.978709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.978686    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:16.978963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:16.978946    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:17.032416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.032383    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:17.032610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.032540    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qxz\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz\") pod \"maas-default-gateway-openshift-default-7d9897b54c-j4f5x\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:17.048404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.048366    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"]
Nov 01 07:08:17.051546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.051522    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1160c6-ktpqb"]
Nov 01 07:08:17.077852 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.077817    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.077852 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.077858    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.077891    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.077946    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078086    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078152    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078176    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078251    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlgn\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078268    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078286    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078456    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078655 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078609    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078775 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078753    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.078894 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.078879    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.080960 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.080938    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.081161 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.081146    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.109195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.109156    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:17.134809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.134779    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.135501 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.135439    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlgn\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn\") pod \"maas-default-gateway-openshift-default-7d9897b54c-zq7gr\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.149271 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.149239    4468 generic.go:356] "Generic (PLEG): container finished" podID="43bf930b-43d5-4357-8ff2-826c5940b15f" containerID="27e7a0966916a932e01749bb7a4ff1928f64096bb0c15f0bc8e9884d4d948e21" exitCode=0
Nov 01 07:08:17.149445 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.149325    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd" event={"ID":"43bf930b-43d5-4357-8ff2-826c5940b15f","Type":"ContainerDied","Data":"27e7a0966916a932e01749bb7a4ff1928f64096bb0c15f0bc8e9884d4d948e21"}
Nov 01 07:08:17.160058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.159949    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:17.545149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.545120    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"]
Nov 01 07:08:17.548657 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:17.548622    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b223fe5_8990_4a0e_bf45_dcff7130ee2c.slice/crio-735022ce51364fa7324f1dacec8fd6e4b32a2954b07d6af6119dac1ec19db9ed WatchSource:0}: Error finding container 735022ce51364fa7324f1dacec8fd6e4b32a2954b07d6af6119dac1ec19db9ed: Status 404 returned error can't find the container with id 735022ce51364fa7324f1dacec8fd6e4b32a2954b07d6af6119dac1ec19db9ed
Nov 01 07:08:17.551998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.551958    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:17.552124 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.552046    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:17.552124 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.552093    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:17.595008 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.594977    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"]
Nov 01 07:08:17.596336 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:17.596286    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfff3421_28a2_4b25_9282_002bcd960b4f.slice/crio-16893c2dcc4f31a5f0f2ca8ddc2f8d8f83cd5580bcf64d8861af80e095cbf9b8 WatchSource:0}: Error finding container 16893c2dcc4f31a5f0f2ca8ddc2f8d8f83cd5580bcf64d8861af80e095cbf9b8: Status 404 returned error can't find the container with id 16893c2dcc4f31a5f0f2ca8ddc2f8d8f83cd5580bcf64d8861af80e095cbf9b8
Nov 01 07:08:17.599508 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.599434    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:17.599631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.599553    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:17.599631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:17.599603    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:08:18.006130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.006102    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 07:08:18.093226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093179    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093257    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093297    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093331    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093355    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093406    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093443    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093507    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093561    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo\") pod \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\" (UID: \"6f86d36a-3c78-44bd-bf20-00e260f66c1e\") "
Nov 01 07:08:18.093842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.093778    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:18.094221 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.094196    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:18.094389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.094364    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:18.094512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.094442    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:18.094612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.094505    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.094612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.094525    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:18.094612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.094531    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.096177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.096134    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:08:18.096286 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.096231    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:18.096286 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.096240    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:18.096540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.096522    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r" pod "6f86d36a-3c78-44bd-bf20-00e260f66c1e" (UID: "6f86d36a-3c78-44bd-bf20-00e260f66c1e"). InnerVolumeSpecName "kube-api-access-btg9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:18.159139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.159040    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" event={"ID":"dfff3421-28a2-4b25-9282-002bcd960b4f","Type":"ContainerStarted","Data":"d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98"}
Nov 01 07:08:18.159139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.159113    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" event={"ID":"dfff3421-28a2-4b25-9282-002bcd960b4f","Type":"ContainerStarted","Data":"16893c2dcc4f31a5f0f2ca8ddc2f8d8f83cd5580bcf64d8861af80e095cbf9b8"}
Nov 01 07:08:18.161611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.161575    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" event={"ID":"0b223fe5-8990-4a0e-bf45-dcff7130ee2c","Type":"ContainerStarted","Data":"103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da"}
Nov 01 07:08:18.161787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.161641    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" event={"ID":"0b223fe5-8990-4a0e-bf45-dcff7130ee2c","Type":"ContainerStarted","Data":"735022ce51364fa7324f1dacec8fd6e4b32a2954b07d6af6119dac1ec19db9ed"}
Nov 01 07:08:18.163685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.163630    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:18.164931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.164895    4468 generic.go:356] "Generic (PLEG): container finished" podID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerID="75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a" exitCode=0
Nov 01 07:08:18.165142 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.165113    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" event={"ID":"6f86d36a-3c78-44bd-bf20-00e260f66c1e","Type":"ContainerDied","Data":"75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a"}
Nov 01 07:08:18.165235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.165172    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf" event={"ID":"6f86d36a-3c78-44bd-bf20-00e260f66c1e","Type":"ContainerDied","Data":"c9b7a57e62e66251d7a4b8c62dfd03aa75919211897960c43a3a5ee77f3cf304"}
Nov 01 07:08:18.165235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.165201    4468 scope.go:122] "RemoveContainer" containerID="75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a"
Nov 01 07:08:18.165235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.165210    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"
Nov 01 07:08:18.165399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.165378    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:18.165454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.165418    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195628    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195664    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195682    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195695    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195710    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btg9r\" (UniqueName: \"kubernetes.io/projected/6f86d36a-3c78-44bd-bf20-00e260f66c1e-kube-api-access-btg9r\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195723    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86d36a-3c78-44bd-bf20-00e260f66c1e-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.195979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.195738    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86d36a-3c78-44bd-bf20-00e260f66c1e-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.206175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.205655    4468 scope.go:122] "RemoveContainer" containerID="84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"
Nov 01 07:08:18.268690 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.268636    4468 scope.go:122] "RemoveContainer" containerID="75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a"
Nov 01 07:08:18.271515 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:18.269057    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a\": container with ID starting with 75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a not found: ID does not exist" containerID="75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a"
Nov 01 07:08:18.271515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.269098    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a"} err="failed to get container status \"75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a\": rpc error: code = NotFound desc = could not find container \"75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a\": container with ID starting with 75bb815054f2fb448d32e5687cf27ea2404456a281dee4baf0f86614cf32562a not found: ID does not exist"
Nov 01 07:08:18.271515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.269123    4468 scope.go:122] "RemoveContainer" containerID="84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"
Nov 01 07:08:18.271515 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:18.269441    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db\": container with ID starting with 84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db not found: ID does not exist" containerID="84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"
Nov 01 07:08:18.271515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.269494    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db"} err="failed to get container status \"84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db\": rpc error: code = NotFound desc = could not find container \"84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db\": container with ID starting with 84a707853ce7ca409b0c319403ff2a37e226f6c3913f69c73ab6b7d6dce475db not found: ID does not exist"
Nov 01 07:08:18.360844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.303544    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podStartSLOduration=2.303523819 podStartE2EDuration="2.303523819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:18.302396578 +0000 UTC m=+3509.870203408" watchObservedRunningTime="2026-11-01 07:08:18.303523819 +0000 UTC m=+3509.871330650"
Nov 01 07:08:18.465414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.465327    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"]
Nov 01 07:08:18.468639 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.468614    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-caf74d-openshift-default-6bd6bf9669-7pnqf"]
Nov 01 07:08:18.497233 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.497206    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:18.565058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.564999    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podStartSLOduration=2.5649810779999997 podStartE2EDuration="2.564981078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:18.560788173 +0000 UTC m=+3510.128594996" watchObservedRunningTime="2026-11-01 07:08:18.564981078 +0000 UTC m=+3510.132787908"
Nov 01 07:08:18.590910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.590875    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" path="/var/lib/kubelet/pods/6f86d36a-3c78-44bd-bf20-00e260f66c1e/volumes"
Nov 01 07:08:18.591336 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.591323    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca52439-a747-4533-b4c7-9c04c77ca8cd" path="/var/lib/kubelet/pods/aca52439-a747-4533-b4c7-9c04c77ca8cd/volumes"
Nov 01 07:08:18.601857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.601825    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca\") pod \"43bf930b-43d5-4357-8ff2-826c5940b15f\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") "
Nov 01 07:08:18.602002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.601877    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7\" (UniqueName: \"kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7\") pod \"43bf930b-43d5-4357-8ff2-826c5940b15f\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") "
Nov 01 07:08:18.602002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.601953    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca\") pod \"43bf930b-43d5-4357-8ff2-826c5940b15f\" (UID: \"43bf930b-43d5-4357-8ff2-826c5940b15f\") "
Nov 01 07:08:18.602271 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.602245    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca" pod "43bf930b-43d5-4357-8ff2-826c5940b15f" (UID: "43bf930b-43d5-4357-8ff2-826c5940b15f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:18.635620 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.611859    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.635984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.635948    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca" pod "43bf930b-43d5-4357-8ff2-826c5940b15f" (UID: "43bf930b-43d5-4357-8ff2-826c5940b15f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:08:18.655539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.655440    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7" pod "43bf930b-43d5-4357-8ff2-826c5940b15f" (UID: "43bf930b-43d5-4357-8ff2-826c5940b15f"). InnerVolumeSpecName "kube-api-access-k9md7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:18.724272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.713548    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9md7\" (UniqueName: \"kubernetes.io/projected/43bf930b-43d5-4357-8ff2-826c5940b15f-kube-api-access-k9md7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:18.724272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:18.713593    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bf930b-43d5-4357-8ff2-826c5940b15f-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:19.115501 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.112862    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:19.139493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.136611    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:19.139493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.136697    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:19.162493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.161879    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:19.162493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.161940    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:19.179141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.179112    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"
Nov 01 07:08:19.179646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.179481    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd" event={"ID":"43bf930b-43d5-4357-8ff2-826c5940b15f","Type":"ContainerDied","Data":"d7037183af1b24a238f083ffcd9b958ba01dd5abaf83e5e89520867425843524"}
Nov 01 07:08:19.179646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.179534    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7037183af1b24a238f083ffcd9b958ba01dd5abaf83e5e89520867425843524"
Nov 01 07:08:19.271155 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:19.271118    4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bf930b_43d5_4357_8ff2_826c5940b15f.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 07:08:19.677604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:19.677571    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"]
Nov 01 07:08:20.125495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:20.111973    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:20.125495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:20.112041    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:20.160722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:20.160621    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:20.160722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:20.160673    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:21.142494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:21.123898    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:21.142494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:21.123969    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:21.160772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:21.160733    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:21.160964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:21.160793    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:22.111057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:22.111014    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:22.111402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:22.111370    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:22.163963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:22.163924    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:22.164452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:22.163984    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:23.111493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:23.110964    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:23.111493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:23.111066    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:23.160609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:23.160509    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:23.160609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:23.160583    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:23.353495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:23.350350    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:08:24.110560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:24.110513    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:24.110793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:24.110577    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:24.161501 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:24.161424    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:24.161720 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:24.161532    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:25.110322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:25.110259    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-j4f5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 01 07:08:25.110926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:25.110353    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 01 07:08:25.161226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:25.161193    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-zq7gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 07:08:25.161436 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:25.161256    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 07:08:26.348398 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:26.348356    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:26.349730 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:26.349698    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:26.350544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:26.350521    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:08:26.362573 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:26.362528    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:26.363482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:26.363432    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:26.367098 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:26.365846    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:08:27.046694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.046660    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"]
Nov 01 07:08:27.049553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.049531    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741cad88-m699g"]
Nov 01 07:08:27.605572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.605536    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"]
Nov 01 07:08:27.607826 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.607798    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy"
Nov 01 07:08:27.607984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.607972    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy"
Nov 01 07:08:27.608153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.608142    4468 cpu_manager.go:414] "containerMap: removing container" podUID="43bf930b-43d5-4357-8ff2-826c5940b15f" containerName="revoke-keys"
Nov 01 07:08:27.608248 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.608237    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bf930b-43d5-4357-8ff2-826c5940b15f" containerName="revoke-keys"
Nov 01 07:08:27.608688 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.608669    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy"
Nov 01 07:08:27.608830 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.608819    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy"
Nov 01 07:08:27.608926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.608917    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bf930b-43d5-4357-8ff2-826c5940b15f" containerName="revoke-keys"
Nov 01 07:08:27.613973 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.613952    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.665859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.665823    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"]
Nov 01 07:08:27.667156 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.667128    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:08:27.710598 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.710557    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:08:27.710793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.710609    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:08:27.710793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.710614    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:08:27.721512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.721460    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zw4p\" (UniqueName: \"kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.721688 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.721561    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.721765 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.721742    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.823252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.823212    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.823444 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.823384    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zw4p\" (UniqueName: \"kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.823517 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.823492    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.824304 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.824276    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.825866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.825847    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.879553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.879450    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zw4p\" (UniqueName: \"kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p\") pod \"maas-api-revoke-sub-24abdc7e-42fnr\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:27.930856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:27.930824    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:28.305374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:28.305343    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"]
Nov 01 07:08:28.306051 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:28.306027    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e98e889_416e_4981_8b2a_bc077c650bc6.slice/crio-c0a72097a3d9c3579e480458682ad2fa4ea47bb055140360b545f9cc49314e7a WatchSource:0}: Error finding container c0a72097a3d9c3579e480458682ad2fa4ea47bb055140360b545f9cc49314e7a: Status 404 returned error can't find the container with id c0a72097a3d9c3579e480458682ad2fa4ea47bb055140360b545f9cc49314e7a
Nov 01 07:08:28.591561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:28.591449    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a47f97-ca09-46af-b4a4-979e27ed3947" path="/var/lib/kubelet/pods/e7a47f97-ca09-46af-b4a4-979e27ed3947/volumes"
Nov 01 07:08:29.291434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:29.291398    4468 generic.go:356] "Generic (PLEG): container finished" podID="9e98e889-416e-4981-8b2a-bc077c650bc6" containerID="7cf5a48d502a77c7d1a55450699297473733b401caa8b381e7447616a02121f6" exitCode=0
Nov 01 07:08:29.291885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:29.291503    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr" event={"ID":"9e98e889-416e-4981-8b2a-bc077c650bc6","Type":"ContainerDied","Data":"7cf5a48d502a77c7d1a55450699297473733b401caa8b381e7447616a02121f6"}
Nov 01 07:08:29.291885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:29.291580    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr" event={"ID":"9e98e889-416e-4981-8b2a-bc077c650bc6","Type":"ContainerStarted","Data":"c0a72097a3d9c3579e480458682ad2fa4ea47bb055140360b545f9cc49314e7a"}
Nov 01 07:08:30.447373 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.447344    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:30.560928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.560838    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca\") pod \"9e98e889-416e-4981-8b2a-bc077c650bc6\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") "
Nov 01 07:08:30.560928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.560907    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p\" (UniqueName: \"kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p\") pod \"9e98e889-416e-4981-8b2a-bc077c650bc6\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") "
Nov 01 07:08:30.561196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.561005    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca\") pod \"9e98e889-416e-4981-8b2a-bc077c650bc6\" (UID: \"9e98e889-416e-4981-8b2a-bc077c650bc6\") "
Nov 01 07:08:30.561258 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.561205    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca" pod "9e98e889-416e-4981-8b2a-bc077c650bc6" (UID: "9e98e889-416e-4981-8b2a-bc077c650bc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:30.561460 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.561440    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:30.565891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.565861    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca" pod "9e98e889-416e-4981-8b2a-bc077c650bc6" (UID: "9e98e889-416e-4981-8b2a-bc077c650bc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:08:30.566003 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.565892    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p" pod "9e98e889-416e-4981-8b2a-bc077c650bc6" (UID: "9e98e889-416e-4981-8b2a-bc077c650bc6"). InnerVolumeSpecName "kube-api-access-4zw4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:30.662700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.662658    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zw4p\" (UniqueName: \"kubernetes.io/projected/9e98e889-416e-4981-8b2a-bc077c650bc6-kube-api-access-4zw4p\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:30.662700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:30.662694    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e98e889-416e-4981-8b2a-bc077c650bc6-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:31.306307 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:31.306269    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"
Nov 01 07:08:31.306529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:31.306315    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr" event={"ID":"9e98e889-416e-4981-8b2a-bc077c650bc6","Type":"ContainerDied","Data":"c0a72097a3d9c3579e480458682ad2fa4ea47bb055140360b545f9cc49314e7a"}
Nov 01 07:08:31.306529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:31.306379    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a72097a3d9c3579e480458682ad2fa4ea47bb055140360b545f9cc49314e7a"
Nov 01 07:08:32.949206 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:32.949170    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:08:34.699189 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:34.699141    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main" containerID="cri-o://f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f" gracePeriod=45
Nov 01 07:08:34.972957 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:34.972928    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 07:08:35.115372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115328    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir\") pod \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") "
Nov 01 07:08:35.115599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115395    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home\") pod \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") "
Nov 01 07:08:35.115599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115492    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2\" (UniqueName: \"kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2\") pod \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") "
Nov 01 07:08:35.115599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115550    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache\") pod \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") "
Nov 01 07:08:35.115776 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115671    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir" pod "4e555b8c-c82b-4d6f-98ff-cc081af6454f" (UID: "4e555b8c-c82b-4d6f-98ff-cc081af6454f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:35.115776 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115714    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs\") pod \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") "
Nov 01 07:08:35.115776 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.115748    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm\") pod \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\" (UID: \"4e555b8c-c82b-4d6f-98ff-cc081af6454f\") "
Nov 01 07:08:35.116159 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.116138    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:35.116397 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.116362    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home" pod "4e555b8c-c82b-4d6f-98ff-cc081af6454f" (UID: "4e555b8c-c82b-4d6f-98ff-cc081af6454f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:35.116575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.116542    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache" pod "4e555b8c-c82b-4d6f-98ff-cc081af6454f" (UID: "4e555b8c-c82b-4d6f-98ff-cc081af6454f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:35.119531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.119507    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm" pod "4e555b8c-c82b-4d6f-98ff-cc081af6454f" (UID: "4e555b8c-c82b-4d6f-98ff-cc081af6454f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:35.119668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.119585    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs" pod "4e555b8c-c82b-4d6f-98ff-cc081af6454f" (UID: "4e555b8c-c82b-4d6f-98ff-cc081af6454f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:08:35.119668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.119613    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2" pod "4e555b8c-c82b-4d6f-98ff-cc081af6454f" (UID: "4e555b8c-c82b-4d6f-98ff-cc081af6454f"). InnerVolumeSpecName "kube-api-access-c6tn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:35.218229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.218140    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:35.218229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.218179    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tn2\" (UniqueName: \"kubernetes.io/projected/4e555b8c-c82b-4d6f-98ff-cc081af6454f-kube-api-access-c6tn2\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:35.218229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.218196    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:35.218229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.218209    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e555b8c-c82b-4d6f-98ff-cc081af6454f-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:35.218229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.218223    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e555b8c-c82b-4d6f-98ff-cc081af6454f-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:35.335786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.335755    4468 generic.go:356] "Generic (PLEG): container finished" podID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerID="f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f" exitCode=0
Nov 01 07:08:35.335978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.335792    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" event={"ID":"4e555b8c-c82b-4d6f-98ff-cc081af6454f","Type":"ContainerDied","Data":"f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f"}
Nov 01 07:08:35.335978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.335809    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"
Nov 01 07:08:35.335978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.335823    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg" event={"ID":"4e555b8c-c82b-4d6f-98ff-cc081af6454f","Type":"ContainerDied","Data":"f51a5d2a6822c17bd3cc3c123dfc3ca897f30b1fe26aae6c9b09791618666ab7"}
Nov 01 07:08:35.335978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.335839    4468 scope.go:122] "RemoveContainer" containerID="f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f"
Nov 01 07:08:35.347433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.347413    4468 scope.go:122] "RemoveContainer" containerID="f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f"
Nov 01 07:08:35.347779 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:35.347760    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f\": container with ID starting with f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f not found: ID does not exist" containerID="f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f"
Nov 01 07:08:35.347838 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.347792    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f"} err="failed to get container status \"f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f\": rpc error: code = NotFound desc = could not find container \"f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f\": container with ID starting with f9d2ae7a8c1e1495cf56cdecd19fd68e98687bd790b2eed14002fbbbce59d24f not found: ID does not exist"
Nov 01 07:08:35.502183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.502097    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"]
Nov 01 07:08:35.551234 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:35.551197    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/premium-w2-8ff537-kserve-68c6579b79-wrrvg"]
Nov 01 07:08:36.589961 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:36.589911    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" path="/var/lib/kubelet/pods/4e555b8c-c82b-4d6f-98ff-cc081af6454f/volumes"
Nov 01 07:08:39.044546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:39.044510    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"]
Nov 01 07:08:39.047645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:39.047619    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5abfaf6-7h7ct"]
Nov 01 07:08:40.590104 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:40.590055    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3840bf-184f-401e-8fb9-52aaf400a0ec" path="/var/lib/kubelet/pods/9d3840bf-184f-401e-8fb9-52aaf400a0ec/volumes"
Nov 01 07:08:49.005901 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:49.005862    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"]
Nov 01 07:08:49.006393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:49.006147    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy" containerID="cri-o://74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c" gracePeriod=30
Nov 01 07:08:50.046047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.046008    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"]
Nov 01 07:08:50.095405 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.095368    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695bf2a1-v2brp"]
Nov 01 07:08:50.154163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154128    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"]
Nov 01 07:08:50.154802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154781    4468 cpu_manager.go:414] "containerMap: removing container" podUID="9e98e889-416e-4981-8b2a-bc077c650bc6" containerName="revoke-keys"
Nov 01 07:08:50.154802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154803    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e98e889-416e-4981-8b2a-bc077c650bc6" containerName="revoke-keys"
Nov 01 07:08:50.154988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154857    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main"
Nov 01 07:08:50.154988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154865    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main"
Nov 01 07:08:50.154988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154880    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy"
Nov 01 07:08:50.154988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.154887    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f86d36a-3c78-44bd-bf20-00e260f66c1e" containerName="istio-proxy"
Nov 01 07:08:50.155192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.155005    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e555b8c-c82b-4d6f-98ff-cc081af6454f" containerName="main"
Nov 01 07:08:50.155192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.155027    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e98e889-416e-4981-8b2a-bc077c650bc6" containerName="revoke-keys"
Nov 01 07:08:50.159904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.159882    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.212459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.212426    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:08:50.212709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.212544    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:08:50.212821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.212544    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:08:50.212821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.212616    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:08:50.215210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.215183    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"]
Nov 01 07:08:50.228033 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.227999    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.228265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.228243    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7bt\" (UniqueName: \"kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.228570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.228536    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.329529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.329424    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7bt\" (UniqueName: \"kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.329691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.329613    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.329691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.329638    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.330358 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.330282    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.332163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.332137    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.372802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.372759    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"]
Nov 01 07:08:50.373092 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.373035    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy" containerID="cri-o://384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288" gracePeriod=30
Nov 01 07:08:50.386081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.386047    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7bt\" (UniqueName: \"kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt\") pod \"maas-api-revoke-sub-a63fb107-dh9pq\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.473963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.473933    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:50.595373 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.595292    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2298205-fee0-4470-885b-bdb15fb85be5" path="/var/lib/kubelet/pods/c2298205-fee0-4470-885b-bdb15fb85be5/volumes"
Nov 01 07:08:50.895940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:50.895911    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"]
Nov 01 07:08:50.897205 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:50.897182    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2ceea1_f3e6_46a4_a7f9_e0fd3b7dd7e2.slice/crio-1f9d01fdfc07d73aef73af960f3eba6bb7edda66563d44d81195c9a75b7018e6 WatchSource:0}: Error finding container 1f9d01fdfc07d73aef73af960f3eba6bb7edda66563d44d81195c9a75b7018e6: Status 404 returned error can't find the container with id 1f9d01fdfc07d73aef73af960f3eba6bb7edda66563d44d81195c9a75b7018e6
Nov 01 07:08:51.421375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.421342    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"]
Nov 01 07:08:51.425454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.425434    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.452859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.452820    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq" event={"ID":"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2","Type":"ContainerStarted","Data":"12c14e4d18a84fbd2c7557f3f0ac620d48764945fbb1391ff02887821bda0f3f"}
Nov 01 07:08:51.453072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.452892    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq" event={"ID":"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2","Type":"ContainerStarted","Data":"1f9d01fdfc07d73aef73af960f3eba6bb7edda66563d44d81195c9a75b7018e6"}
Nov 01 07:08:51.482165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.482087    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"]
Nov 01 07:08:51.551211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.551166    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.551423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.551266    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbdf\" (UniqueName: \"kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.551423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.551357    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.651085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.651025    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq" podStartSLOduration=1.651008604 podStartE2EDuration="1.651008604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:51.649784532 +0000 UTC m=+3543.217591362" watchObservedRunningTime="2026-11-01 07:08:51.651008604 +0000 UTC m=+3543.218815457"
Nov 01 07:08:51.652936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.652905    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.653099 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.653053    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.653165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.653099    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbdf\" (UniqueName: \"kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.653766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.653744    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.655492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.655452    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.710518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.710486    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbdf\" (UniqueName: \"kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf\") pod \"maas-api-revoke-sub-1f689c3d-xslfj\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:51.739540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:51.739459    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:52.115594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.115561    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"]
Nov 01 07:08:52.116684 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:08:52.116657    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744cd94b_5f35_4b48_8caf_905b8f17890c.slice/crio-971eba7282f63aa7bb70b80bf5abac26036b4cba3a0758c3e0f6011362ac6cca WatchSource:0}: Error finding container 971eba7282f63aa7bb70b80bf5abac26036b4cba3a0758c3e0f6011362ac6cca: Status 404 returned error can't find the container with id 971eba7282f63aa7bb70b80bf5abac26036b4cba3a0758c3e0f6011362ac6cca
Nov 01 07:08:52.171798 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.171762    4468 scope.go:122] "RemoveContainer" containerID="44f14f24cffd6153f5439ac8eb5324d721087c377cf4ac7af7c9a954832f253f"
Nov 01 07:08:52.184853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.184827    4468 scope.go:122] "RemoveContainer" containerID="7b7d728ae67d39570aab9a7b052eeb4b812e9c437ef350e1d7f64cddcaad0934"
Nov 01 07:08:52.461192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.461155    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj" event={"ID":"744cd94b-5f35-4b48-8caf-905b8f17890c","Type":"ContainerStarted","Data":"4a1c972fd533981f10d07809de0e7c7edb911d1941a44bbc02258a042e2f4766"}
Nov 01 07:08:52.461697 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.461209    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj" event={"ID":"744cd94b-5f35-4b48-8caf-905b8f17890c","Type":"ContainerStarted","Data":"971eba7282f63aa7bb70b80bf5abac26036b4cba3a0758c3e0f6011362ac6cca"}
Nov 01 07:08:52.462918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.462887    4468 generic.go:356] "Generic (PLEG): container finished" podID="ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" containerID="12c14e4d18a84fbd2c7557f3f0ac620d48764945fbb1391ff02887821bda0f3f" exitCode=0
Nov 01 07:08:52.463041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.462967    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq" event={"ID":"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2","Type":"ContainerDied","Data":"12c14e4d18a84fbd2c7557f3f0ac620d48764945fbb1391ff02887821bda0f3f"}
Nov 01 07:08:52.575913 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:52.575333    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj" podStartSLOduration=1.575314637 podStartE2EDuration="1.575314637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:52.574430759 +0000 UTC m=+3544.142237627" watchObservedRunningTime="2026-11-01 07:08:52.575314637 +0000 UTC m=+3544.143121472"
Nov 01 07:08:53.470927 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.470890    4468 generic.go:356] "Generic (PLEG): container finished" podID="744cd94b-5f35-4b48-8caf-905b8f17890c" containerID="4a1c972fd533981f10d07809de0e7c7edb911d1941a44bbc02258a042e2f4766" exitCode=0
Nov 01 07:08:53.471429 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.470975    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj" event={"ID":"744cd94b-5f35-4b48-8caf-905b8f17890c","Type":"ContainerDied","Data":"4a1c972fd533981f10d07809de0e7c7edb911d1941a44bbc02258a042e2f4766"}
Nov 01 07:08:53.626483 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.626441    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:53.677338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.677305    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca\") pod \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") "
Nov 01 07:08:53.677571 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.677553    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt\" (UniqueName: \"kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt\") pod \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") "
Nov 01 07:08:53.677678 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.677663    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca\") pod \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\" (UID: \"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2\") "
Nov 01 07:08:53.677745 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.677711    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca" pod "ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" (UID: "ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:53.678112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.678091    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:53.679866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.679839    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt" pod "ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" (UID: "ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2"). InnerVolumeSpecName "kube-api-access-5w7bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:53.679866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.679857    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca" pod "ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" (UID: "ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:08:53.779871 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.779786    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:53.779871 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:53.779823    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7bt\" (UniqueName: \"kubernetes.io/projected/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2-kube-api-access-5w7bt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.271252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.271225    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 07:08:54.385805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.385759    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.385832    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.385901    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.385925    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.385962    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.386046    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.386159    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.386194    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.386260    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs\") pod \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\" (UID: \"b3bc3aa9-90af-4e8c-bb37-e614ca82d170\") "
Nov 01 07:08:54.386873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.386825    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:54.386873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.386842    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:54.387188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387056    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:54.387188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387089    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:54.387324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387287    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.387324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387307    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.387428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387324    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.387428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387339    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.387609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.387588    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:54.389134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.389110    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:08:54.389244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.389179    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "kube-api-access-6nnlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:54.389394 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.389376    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:54.389503 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.389427    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token" pod "b3bc3aa9-90af-4e8c-bb37-e614ca82d170" (UID: "b3bc3aa9-90af-4e8c-bb37-e614ca82d170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:54.480225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.480169    4468 generic.go:356] "Generic (PLEG): container finished" podID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerID="74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c" exitCode=0
Nov 01 07:08:54.480741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.480230    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"
Nov 01 07:08:54.480741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.480260    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" event={"ID":"b3bc3aa9-90af-4e8c-bb37-e614ca82d170","Type":"ContainerDied","Data":"74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c"}
Nov 01 07:08:54.480741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.480327    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5" event={"ID":"b3bc3aa9-90af-4e8c-bb37-e614ca82d170","Type":"ContainerDied","Data":"0c50c5ac4e5dbf90d503d960aff94b3fe7e477943228ec4cd17f0bc0c71e86f2"}
Nov 01 07:08:54.480741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.480362    4468 scope.go:122] "RemoveContainer" containerID="74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c"
Nov 01 07:08:54.483374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.483348    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"
Nov 01 07:08:54.483526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.483416    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq" event={"ID":"ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2","Type":"ContainerDied","Data":"1f9d01fdfc07d73aef73af960f3eba6bb7edda66563d44d81195c9a75b7018e6"}
Nov 01 07:08:54.483526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.483460    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f9d01fdfc07d73aef73af960f3eba6bb7edda66563d44d81195c9a75b7018e6"
Nov 01 07:08:54.488070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.488047    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nnlw\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-kube-api-access-6nnlw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.488070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.488070    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.488261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.488082    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.488261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.488092    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.488261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.488101    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bc3aa9-90af-4e8c-bb37-e614ca82d170-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.494011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.493987    4468 scope.go:122] "RemoveContainer" containerID="5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"
Nov 01 07:08:54.504868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.504846    4468 scope.go:122] "RemoveContainer" containerID="74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c"
Nov 01 07:08:54.505212 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:54.505186    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c\": container with ID starting with 74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c not found: ID does not exist" containerID="74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c"
Nov 01 07:08:54.505344 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.505221    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c"} err="failed to get container status \"74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c\": rpc error: code = NotFound desc = could not find container \"74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c\": container with ID starting with 74fa54da37a93bd65a75c6ebd9f069cb77fc8ea6fed971bb99812bc87efe1f7c not found: ID does not exist"
Nov 01 07:08:54.505344 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.505240    4468 scope.go:122] "RemoveContainer" containerID="5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"
Nov 01 07:08:54.505562 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:54.505532    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc\": container with ID starting with 5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc not found: ID does not exist" containerID="5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"
Nov 01 07:08:54.505644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.505559    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc"} err="failed to get container status \"5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc\": rpc error: code = NotFound desc = could not find container \"5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc\": container with ID starting with 5c2b17abf3dfecd6417f480c54e36063e856ca474768757a92e3c08c156c7bdc not found: ID does not exist"
Nov 01 07:08:54.633981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.633955    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:54.644956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.644878    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"]
Nov 01 07:08:54.647823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.647801    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6ba6e5-openshift-default-56cbff86ff-hrkr5"]
Nov 01 07:08:54.689802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.689767    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca\") pod \"744cd94b-5f35-4b48-8caf-905b8f17890c\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") "
Nov 01 07:08:54.690300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.689888    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca\") pod \"744cd94b-5f35-4b48-8caf-905b8f17890c\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") "
Nov 01 07:08:54.690300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.690032    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf\" (UniqueName: \"kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf\") pod \"744cd94b-5f35-4b48-8caf-905b8f17890c\" (UID: \"744cd94b-5f35-4b48-8caf-905b8f17890c\") "
Nov 01 07:08:54.690433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.690326    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca" pod "744cd94b-5f35-4b48-8caf-905b8f17890c" (UID: "744cd94b-5f35-4b48-8caf-905b8f17890c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:54.690644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.690589    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.692629 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.692607    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca" pod "744cd94b-5f35-4b48-8caf-905b8f17890c" (UID: "744cd94b-5f35-4b48-8caf-905b8f17890c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:08:54.692740 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.692675    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf" pod "744cd94b-5f35-4b48-8caf-905b8f17890c" (UID: "744cd94b-5f35-4b48-8caf-905b8f17890c"). InnerVolumeSpecName "kube-api-access-trbdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:54.791578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.791548    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trbdf\" (UniqueName: \"kubernetes.io/projected/744cd94b-5f35-4b48-8caf-905b8f17890c-kube-api-access-trbdf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:54.791578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:54.791575    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744cd94b-5f35-4b48-8caf-905b8f17890c-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.491836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.491803    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj" event={"ID":"744cd94b-5f35-4b48-8caf-905b8f17890c","Type":"ContainerDied","Data":"971eba7282f63aa7bb70b80bf5abac26036b4cba3a0758c3e0f6011362ac6cca"}
Nov 01 07:08:55.492306 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.491844    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"
Nov 01 07:08:55.492306 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.491853    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971eba7282f63aa7bb70b80bf5abac26036b4cba3a0758c3e0f6011362ac6cca"
Nov 01 07:08:55.654107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.654041    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 07:08:55.804206 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804168    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804231    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804273    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804398    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804649 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804442    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804649 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804501    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804649 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804560    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804649 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804609    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.804865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804674    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo\") pod \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\" (UID: \"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc\") "
Nov 01 07:08:55.805755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804497    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:55.805755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804689    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:55.805755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.804892    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:55.805755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.805309    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:55.805755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.805722    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:08:55.808868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.808777    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:08:55.808868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.808823    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "kube-api-access-kcb5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:55.808868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.808847    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:55.811441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.811414    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token" pod "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" (UID: "4147bbc5-e12f-45e2-8f0e-87f046fc8bdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908162    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908200    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908212    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908224    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908236    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcb5z\" (UniqueName: \"kubernetes.io/projected/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-kube-api-access-kcb5z\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908249    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908260    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908271    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.908365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.908282    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:55.953319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:55.953272    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"]
Nov 01 07:08:56.500437 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.500398    4468 generic.go:356] "Generic (PLEG): container finished" podID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerID="384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288" exitCode=0
Nov 01 07:08:56.500940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.500455    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"
Nov 01 07:08:56.500940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.500504    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" event={"ID":"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc","Type":"ContainerDied","Data":"384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288"}
Nov 01 07:08:56.500940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.500600    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687" event={"ID":"4147bbc5-e12f-45e2-8f0e-87f046fc8bdc","Type":"ContainerDied","Data":"aea54d618ccc54c2cbf5b5e9eb489fa12115597ef0d795925d41f84a70e86da2"}
Nov 01 07:08:56.500940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.500629    4468 scope.go:122] "RemoveContainer" containerID="384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288"
Nov 01 07:08:56.511793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.511771    4468 scope.go:122] "RemoveContainer" containerID="007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"
Nov 01 07:08:56.521442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.521425    4468 scope.go:122] "RemoveContainer" containerID="384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288"
Nov 01 07:08:56.521707 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:56.521691    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288\": container with ID starting with 384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288 not found: ID does not exist" containerID="384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288"
Nov 01 07:08:56.521753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.521717    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288"} err="failed to get container status \"384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288\": rpc error: code = NotFound desc = could not find container \"384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288\": container with ID starting with 384232c15dd3d8e960317825c9df8cce4badb4fc0d2850ea7c3f60a76df3b288 not found: ID does not exist"
Nov 01 07:08:56.521753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.521736    4468 scope.go:122] "RemoveContainer" containerID="007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"
Nov 01 07:08:56.521979 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:08:56.521961    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446\": container with ID starting with 007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446 not found: ID does not exist" containerID="007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"
Nov 01 07:08:56.522022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.521985    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446"} err="failed to get container status \"007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446\": rpc error: code = NotFound desc = could not find container \"007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446\": container with ID starting with 007bbd840eccf2b6bd9632a8053ec2981cba055fb1489f9b2a9f15f12a760446 not found: ID does not exist"
Nov 01 07:08:56.591068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.591031    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" path="/var/lib/kubelet/pods/b3bc3aa9-90af-4e8c-bb37-e614ca82d170/volumes"
Nov 01 07:08:56.665226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.665189    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"]
Nov 01 07:08:56.714399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:56.714363    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-581487-openshift-default-755f5b8d9c-b5687"]
Nov 01 07:08:58.589907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:08:58.589872    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" path="/var/lib/kubelet/pods/4147bbc5-e12f-45e2-8f0e-87f046fc8bdc/volumes"
Nov 01 07:09:01.250707 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:01.250588    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"]
Nov 01 07:09:02.575797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.575754    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"]
Nov 01 07:09:02.576505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576459    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy"
Nov 01 07:09:02.576505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576502    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576519    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576526    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576545    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" containerName="revoke-keys"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576552    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" containerName="revoke-keys"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576562    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576568    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576604    4468 cpu_manager.go:414] "containerMap: removing container" podUID="744cd94b-5f35-4b48-8caf-905b8f17890c" containerName="revoke-keys"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576611    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744cd94b-5f35-4b48-8caf-905b8f17890c" containerName="revoke-keys"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576628    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy"
Nov 01 07:09:02.576670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576636    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy"
Nov 01 07:09:02.577165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576777    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy"
Nov 01 07:09:02.577165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576794    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4147bbc5-e12f-45e2-8f0e-87f046fc8bdc" containerName="istio-proxy"
Nov 01 07:09:02.577165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576804    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" containerName="revoke-keys"
Nov 01 07:09:02.577165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576813    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy"
Nov 01 07:09:02.577165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.576828    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="744cd94b-5f35-4b48-8caf-905b8f17890c" containerName="revoke-keys"
Nov 01 07:09:02.581962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.581938    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.633952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.633909    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:09:02.634151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.633958    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:09:02.634151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.633964    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:09:02.683738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.683701    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:09:02.688867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.688827    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"]
Nov 01 07:09:02.693716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.693659    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg56n\" (UniqueName: \"kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.697190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.697158    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.698175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.698152    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.799722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.799680    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.799934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.799735    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg56n\" (UniqueName: \"kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.799934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.799883    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.800345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.800324    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.802332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.802314    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.856899 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.856814    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg56n\" (UniqueName: \"kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n\") pod \"maas-api-revoke-sub-1eae3943-krfqr\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.895528 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:02.895484    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:02.950478 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:02.950427    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:09:03.276493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:03.276450    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"]
Nov 01 07:09:03.278297 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:09:03.278268    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e882d15_b551_44cc_b76d_74cf56600887.slice/crio-6370e517ae24c47c0ad4013224f7bc39c47e37f347d7b3a45c4fdaebe21198bb WatchSource:0}: Error finding container 6370e517ae24c47c0ad4013224f7bc39c47e37f347d7b3a45c4fdaebe21198bb: Status 404 returned error can't find the container with id 6370e517ae24c47c0ad4013224f7bc39c47e37f347d7b3a45c4fdaebe21198bb
Nov 01 07:09:03.548162 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:03.548128    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr" event={"ID":"3e882d15-b551-44cc-b76d-74cf56600887","Type":"ContainerStarted","Data":"6370e517ae24c47c0ad4013224f7bc39c47e37f347d7b3a45c4fdaebe21198bb"}
Nov 01 07:09:04.532272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.532186    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"]
Nov 01 07:09:04.532886 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.532870    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bc3aa9-90af-4e8c-bb37-e614ca82d170" containerName="istio-proxy"
Nov 01 07:09:04.536044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.536028    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.555677 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.555640    4468 generic.go:356] "Generic (PLEG): container finished" podID="3e882d15-b551-44cc-b76d-74cf56600887" containerID="4a6386f71ad90d4ab72d2e3266daeda098651f3cdda57757dd5cc6120cb8f8bb" exitCode=0
Nov 01 07:09:04.555849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.555723    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr" event={"ID":"3e882d15-b551-44cc-b76d-74cf56600887","Type":"ContainerDied","Data":"4a6386f71ad90d4ab72d2e3266daeda098651f3cdda57757dd5cc6120cb8f8bb"}
Nov 01 07:09:04.605976 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.605936    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"]
Nov 01 07:09:04.629509 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.629453    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:09:04.721447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.721408    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.721648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.721501    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txx5f\" (UniqueName: \"kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.721648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.721627    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.822837 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.822799    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.823065 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.822857    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txx5f\" (UniqueName: \"kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.823065 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.822924    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.823963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.823931    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.825612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.825585    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:04.880628 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:04.880539    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txx5f\" (UniqueName: \"kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f\") pod \"maas-api-revoke-sub-8a3fce63-86sn7\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:05.150267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.150177    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:05.583701 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:09:05.583672    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bcbb8e_44a4_4fb8_b0a8_3a61c72b7a7f.slice/crio-5c4de3bbc028555821e3fc231d7311dd73cdb7e3ac532c4b172a5a98d6b32a34 WatchSource:0}: Error finding container 5c4de3bbc028555821e3fc231d7311dd73cdb7e3ac532c4b172a5a98d6b32a34: Status 404 returned error can't find the container with id 5c4de3bbc028555821e3fc231d7311dd73cdb7e3ac532c4b172a5a98d6b32a34
Nov 01 07:09:05.584046 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.584023    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"]
Nov 01 07:09:05.855715 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.851177    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:05.973355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.973324    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca\") pod \"3e882d15-b551-44cc-b76d-74cf56600887\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") "
Nov 01 07:09:05.973611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.973593    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n\" (UniqueName: \"kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n\") pod \"3e882d15-b551-44cc-b76d-74cf56600887\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") "
Nov 01 07:09:05.973715 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.973692    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca\") pod \"3e882d15-b551-44cc-b76d-74cf56600887\" (UID: \"3e882d15-b551-44cc-b76d-74cf56600887\") "
Nov 01 07:09:05.974097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.974066    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca" pod "3e882d15-b551-44cc-b76d-74cf56600887" (UID: "3e882d15-b551-44cc-b76d-74cf56600887"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:05.974319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.974265    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:05.976812 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.976732    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca" pod "3e882d15-b551-44cc-b76d-74cf56600887" (UID: "3e882d15-b551-44cc-b76d-74cf56600887"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:09:05.976812 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:05.976789    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n" pod "3e882d15-b551-44cc-b76d-74cf56600887" (UID: "3e882d15-b551-44cc-b76d-74cf56600887"). InnerVolumeSpecName "kube-api-access-xg56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:06.049129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.049087    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"]
Nov 01 07:09:06.075024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.074986    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg56n\" (UniqueName: \"kubernetes.io/projected/3e882d15-b551-44cc-b76d-74cf56600887-kube-api-access-xg56n\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:06.075024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.075018    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e882d15-b551-44cc-b76d-74cf56600887-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:06.166220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.166174    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"]
Nov 01 07:09:06.251653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.251605    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerName="payload-processing" containerID="cri-o://2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3" gracePeriod=25
Nov 01 07:09:06.576575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.576534    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7" event={"ID":"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f","Type":"ContainerStarted","Data":"a9e6f7bd083e4b4eaf81232a76ea9191096dbb5b2dd756d56032d08f85f07d3f"}
Nov 01 07:09:06.576786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.576597    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7" event={"ID":"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f","Type":"ContainerStarted","Data":"5c4de3bbc028555821e3fc231d7311dd73cdb7e3ac532c4b172a5a98d6b32a34"}
Nov 01 07:09:06.577939 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.577920    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"
Nov 01 07:09:06.578019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.577959    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr" event={"ID":"3e882d15-b551-44cc-b76d-74cf56600887","Type":"ContainerDied","Data":"6370e517ae24c47c0ad4013224f7bc39c47e37f347d7b3a45c4fdaebe21198bb"}
Nov 01 07:09:06.578019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.577997    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6370e517ae24c47c0ad4013224f7bc39c47e37f347d7b3a45c4fdaebe21198bb"
Nov 01 07:09:06.693486 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:06.693409    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7" podStartSLOduration=2.693394414 podStartE2EDuration="2.693394414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:06.690511491 +0000 UTC m=+3558.258318323" watchObservedRunningTime="2026-11-01 07:09:06.693394414 +0000 UTC m=+3558.261201243"
Nov 01 07:09:07.586044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:07.586006    4468 generic.go:356] "Generic (PLEG): container finished" podID="80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" containerID="a9e6f7bd083e4b4eaf81232a76ea9191096dbb5b2dd756d56032d08f85f07d3f" exitCode=0
Nov 01 07:09:07.586247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:07.586093    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7" event={"ID":"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f","Type":"ContainerDied","Data":"a9e6f7bd083e4b4eaf81232a76ea9191096dbb5b2dd756d56032d08f85f07d3f"}
Nov 01 07:09:08.741542 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.741514    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:08.804586 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.804554    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f\" (UniqueName: \"kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f\") pod \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") "
Nov 01 07:09:08.804807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.804624    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca\") pod \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") "
Nov 01 07:09:08.804807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.804775    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca\") pod \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\" (UID: \"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f\") "
Nov 01 07:09:08.805053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.805028    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca" pod "80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" (UID: "80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:08.806861 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.806839    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f" pod "80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" (UID: "80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f"). InnerVolumeSpecName "kube-api-access-txx5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:08.807004 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.806982    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca" pod "80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" (UID: "80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:09:08.905964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.905878    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txx5f\" (UniqueName: \"kubernetes.io/projected/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-kube-api-access-txx5f\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:08.905964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.905908    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:08.905964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:08.905920    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:09.609920 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:09.609891    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"
Nov 01 07:09:09.610100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:09.609910    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7" event={"ID":"80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f","Type":"ContainerDied","Data":"5c4de3bbc028555821e3fc231d7311dd73cdb7e3ac532c4b172a5a98d6b32a34"}
Nov 01 07:09:09.610100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:09.609991    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4de3bbc028555821e3fc231d7311dd73cdb7e3ac532c4b172a5a98d6b32a34"
Nov 01 07:09:10.975142 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:10.975096    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main" containerID="cri-o://411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95" gracePeriod=45
Nov 01 07:09:11.049934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.049885    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerName="payload-processing" containerID="cri-o://4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410" gracePeriod=25
Nov 01 07:09:11.166544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.166493    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerName="payload-pre-processing" containerID="cri-o://e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9" gracePeriod=25
Nov 01 07:09:11.253294 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.253266    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 07:09:11.336227 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336190    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77\" (UniqueName: \"kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77\") pod \"30617666-577d-44e0-9961-1e4978434e76\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") "
Nov 01 07:09:11.336420 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336378    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm\") pod \"30617666-577d-44e0-9961-1e4978434e76\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") "
Nov 01 07:09:11.336499 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336433    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home\") pod \"30617666-577d-44e0-9961-1e4978434e76\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") "
Nov 01 07:09:11.336499 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336456    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs\" (UniqueName: \"kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs\") pod \"30617666-577d-44e0-9961-1e4978434e76\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") "
Nov 01 07:09:11.336612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336525    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir\") pod \"30617666-577d-44e0-9961-1e4978434e76\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") "
Nov 01 07:09:11.336612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336548    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache\") pod \"30617666-577d-44e0-9961-1e4978434e76\" (UID: \"30617666-577d-44e0-9961-1e4978434e76\") "
Nov 01 07:09:11.336800 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336749    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home" pod "30617666-577d-44e0-9961-1e4978434e76" (UID: "30617666-577d-44e0-9961-1e4978434e76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:11.337088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336812    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir" pod "30617666-577d-44e0-9961-1e4978434e76" (UID: "30617666-577d-44e0-9961-1e4978434e76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:11.337088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.336900    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache" pod "30617666-577d-44e0-9961-1e4978434e76" (UID: "30617666-577d-44e0-9961-1e4978434e76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:11.338884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.338857    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs" pod "30617666-577d-44e0-9961-1e4978434e76" (UID: "30617666-577d-44e0-9961-1e4978434e76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:09:11.338884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.338867    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm" pod "30617666-577d-44e0-9961-1e4978434e76" (UID: "30617666-577d-44e0-9961-1e4978434e76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:11.339011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.338892    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77" pod "30617666-577d-44e0-9961-1e4978434e76" (UID: "30617666-577d-44e0-9961-1e4978434e76"). InnerVolumeSpecName "kube-api-access-6nn77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:11.438578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.438532    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:11.438578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.438574    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:11.438578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.438583    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30617666-577d-44e0-9961-1e4978434e76-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:11.438819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.438612    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:11.438819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.438621    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30617666-577d-44e0-9961-1e4978434e76-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:11.438819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.438632    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nn77\" (UniqueName: \"kubernetes.io/projected/30617666-577d-44e0-9961-1e4978434e76-kube-api-access-6nn77\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:11.520527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.520392    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:11.521077 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.521047    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:11.623859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.623827    4468 generic.go:356] "Generic (PLEG): container finished" podID="30617666-577d-44e0-9961-1e4978434e76" containerID="411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95" exitCode=0
Nov 01 07:09:11.624021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.623870    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" event={"ID":"30617666-577d-44e0-9961-1e4978434e76","Type":"ContainerDied","Data":"411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95"}
Nov 01 07:09:11.624021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.623887    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"
Nov 01 07:09:11.624021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.623904    4468 scope.go:122] "RemoveContainer" containerID="411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95"
Nov 01 07:09:11.624021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.623895    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc" event={"ID":"30617666-577d-44e0-9961-1e4978434e76","Type":"ContainerDied","Data":"969bc4302219a4cf5d7ac3fc5e35a9a3dc37fdf5835b9e03a124e2d8c06bfdc9"}
Nov 01 07:09:11.634706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.634685    4468 scope.go:122] "RemoveContainer" containerID="411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95"
Nov 01 07:09:11.635040 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:11.635021    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95\": container with ID starting with 411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95 not found: ID does not exist" containerID="411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95"
Nov 01 07:09:11.635095 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.635049    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95"} err="failed to get container status \"411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95\": rpc error: code = NotFound desc = could not find container \"411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95\": container with ID starting with 411af6de003ef81d49c75fa9aee20972e6fe414ce180ab4e4641d38c2bbb3a95 not found: ID does not exist"
Nov 01 07:09:11.786624 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.786521    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"]
Nov 01 07:09:11.790344 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:11.790314    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-2-w4-190724-kserve-5fbdb77c67-ns7hc"]
Nov 01 07:09:12.524889 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:12.524851    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:12.590040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:12.590001    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30617666-577d-44e0-9961-1e4978434e76" path="/var/lib/kubelet/pods/30617666-577d-44e0-9961-1e4978434e76/volumes"
Nov 01 07:09:18.947369 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947328    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"]
Nov 01 07:09:18.947936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947909    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3e882d15-b551-44cc-b76d-74cf56600887" containerName="revoke-keys"
Nov 01 07:09:18.947936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947928    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e882d15-b551-44cc-b76d-74cf56600887" containerName="revoke-keys"
Nov 01 07:09:18.947936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947941    4468 cpu_manager.go:414] "containerMap: removing container" podUID="80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" containerName="revoke-keys"
Nov 01 07:09:18.948134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947946    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" containerName="revoke-keys"
Nov 01 07:09:18.948134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947960    4468 cpu_manager.go:414] "containerMap: removing container" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main"
Nov 01 07:09:18.948134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.947966    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main"
Nov 01 07:09:18.948134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.948051    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e882d15-b551-44cc-b76d-74cf56600887" containerName="revoke-keys"
Nov 01 07:09:18.948134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.948061    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" containerName="revoke-keys"
Nov 01 07:09:18.948134 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.948068    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="30617666-577d-44e0-9961-1e4978434e76" containerName="main"
Nov 01 07:09:18.952591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:18.952574    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.008402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.008369    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"]
Nov 01 07:09:19.028618 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.028586    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.028828 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.028628    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xlj\" (UniqueName: \"kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.028828 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.028707    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.054100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.054064    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:09:19.054100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.054064    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:09:19.054413 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.054398    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:09:19.054533 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.054438    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:09:19.130557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.130518    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.130708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.130574    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xlj\" (UniqueName: \"kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.130708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.130603    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.131336 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.131317    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.133145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.133118    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.187039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.186999    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xlj\" (UniqueName: \"kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj\") pod \"maas-api-revoke-sub-3e416163-vpsxp\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.265430 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.265342    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:19.638066 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.638040    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"]
Nov 01 07:09:19.639168 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:09:19.639141    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a12e3c_6346_46ca_ad37_55dac1c345c6.slice/crio-c11a94f88c34455326e0ce58d965f2ffa269829c3ad634fa50be8effd8d02163 WatchSource:0}: Error finding container c11a94f88c34455326e0ce58d965f2ffa269829c3ad634fa50be8effd8d02163: Status 404 returned error can't find the container with id c11a94f88c34455326e0ce58d965f2ffa269829c3ad634fa50be8effd8d02163
Nov 01 07:09:19.677693 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:19.677660    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp" event={"ID":"00a12e3c-6346-46ca-ad37-55dac1c345c6","Type":"ContainerStarted","Data":"c11a94f88c34455326e0ce58d965f2ffa269829c3ad634fa50be8effd8d02163"}
Nov 01 07:09:20.686308 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:20.686271    4468 generic.go:356] "Generic (PLEG): container finished" podID="00a12e3c-6346-46ca-ad37-55dac1c345c6" containerID="5f4b1c3159ede3797ff735374d3d6a55ec2b9eb02f8ba383070b8bee1ee002d1" exitCode=0
Nov 01 07:09:20.686805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:20.686357    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp" event={"ID":"00a12e3c-6346-46ca-ad37-55dac1c345c6","Type":"ContainerDied","Data":"5f4b1c3159ede3797ff735374d3d6a55ec2b9eb02f8ba383070b8bee1ee002d1"}
Nov 01 07:09:21.520284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.520247    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:21.520575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.520260    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:21.836199 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.836172    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:21.964228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.964185    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca\") pod \"00a12e3c-6346-46ca-ad37-55dac1c345c6\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") "
Nov 01 07:09:21.964438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.964242    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca\") pod \"00a12e3c-6346-46ca-ad37-55dac1c345c6\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") "
Nov 01 07:09:21.964438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.964346    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj\" (UniqueName: \"kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj\") pod \"00a12e3c-6346-46ca-ad37-55dac1c345c6\" (UID: \"00a12e3c-6346-46ca-ad37-55dac1c345c6\") "
Nov 01 07:09:21.964723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.964686    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca" pod "00a12e3c-6346-46ca-ad37-55dac1c345c6" (UID: "00a12e3c-6346-46ca-ad37-55dac1c345c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:21.965027 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.965006    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:21.966767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.966741    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj" pod "00a12e3c-6346-46ca-ad37-55dac1c345c6" (UID: "00a12e3c-6346-46ca-ad37-55dac1c345c6"). InnerVolumeSpecName "kube-api-access-m5xlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:21.966885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:21.966806    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca" pod "00a12e3c-6346-46ca-ad37-55dac1c345c6" (UID: "00a12e3c-6346-46ca-ad37-55dac1c345c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:09:22.066370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:22.066265    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a12e3c-6346-46ca-ad37-55dac1c345c6-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:22.066370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:22.066313    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xlj\" (UniqueName: \"kubernetes.io/projected/00a12e3c-6346-46ca-ad37-55dac1c345c6-kube-api-access-m5xlj\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:22.524710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:22.524618    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:09:22.713222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:22.711676    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp" event={"ID":"00a12e3c-6346-46ca-ad37-55dac1c345c6","Type":"ContainerDied","Data":"c11a94f88c34455326e0ce58d965f2ffa269829c3ad634fa50be8effd8d02163"}
Nov 01 07:09:22.713222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:22.711741    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c11a94f88c34455326e0ce58d965f2ffa269829c3ad634fa50be8effd8d02163"
Nov 01 07:09:22.716564 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:22.713578    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"
Nov 01 07:09:26.515630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.515572    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 07:09:26.616060 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.616025    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt\" (UniqueName: \"kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt\") pod \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") "
Nov 01 07:09:26.616230 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.616130    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume\") pod \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\" (UID: \"bf29ea2f-d691-4df5-af90-4ff4cd63ba33\") "
Nov 01 07:09:26.616519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.616496    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume" pod "bf29ea2f-d691-4df5-af90-4ff4cd63ba33" (UID: "bf29ea2f-d691-4df5-af90-4ff4cd63ba33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:26.618245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.618227    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt" pod "bf29ea2f-d691-4df5-af90-4ff4cd63ba33" (UID: "bf29ea2f-d691-4df5-af90-4ff4cd63ba33"). InnerVolumeSpecName "kube-api-access-zcfbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:26.717949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.717915    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:26.717949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.717942    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfbt\" (UniqueName: \"kubernetes.io/projected/bf29ea2f-d691-4df5-af90-4ff4cd63ba33-kube-api-access-zcfbt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:26.738442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.738408    4468 generic.go:356] "Generic (PLEG): container finished" podID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerID="2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3" exitCode=0
Nov 01 07:09:26.738644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.738456    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"
Nov 01 07:09:26.738644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.738500    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" event={"ID":"bf29ea2f-d691-4df5-af90-4ff4cd63ba33","Type":"ContainerDied","Data":"2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3"}
Nov 01 07:09:26.738644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.738575    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb" event={"ID":"bf29ea2f-d691-4df5-af90-4ff4cd63ba33","Type":"ContainerDied","Data":"abb956eff951e07ed8ec5bbec4902c6652262ca97a1bbec9e06907cebf6e8aa2"}
Nov 01 07:09:26.738644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.738607    4468 scope.go:122] "RemoveContainer" containerID="2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3"
Nov 01 07:09:26.748695 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.748635    4468 scope.go:122] "RemoveContainer" containerID="2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3"
Nov 01 07:09:26.748945 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:26.748923    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3\": container with ID starting with 2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3 not found: ID does not exist" containerID="2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3"
Nov 01 07:09:26.749002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.748954    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3"} err="failed to get container status \"2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3\": rpc error: code = NotFound desc = could not find container \"2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3\": container with ID starting with 2abba44baaf5d08579b349d557b6d1133a3df7f59cfe47cbb09448bf0b6d58b3 not found: ID does not exist"
Nov 01 07:09:26.904160 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.904126    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"]
Nov 01 07:09:26.909407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:26.909381    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3ba2f3-5b69fb897-6bmzb"]
Nov 01 07:09:28.590366 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:28.590333    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" path="/var/lib/kubelet/pods/bf29ea2f-d691-4df5-af90-4ff4cd63ba33/volumes"
Nov 01 07:09:31.351349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.351324    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 07:09:31.368830 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.368800    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4\" (UniqueName: \"kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4\") pod \"fdbec8c6-4548-4f51-b683-2864ea8088a5\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") "
Nov 01 07:09:31.369371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.369017    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume\") pod \"fdbec8c6-4548-4f51-b683-2864ea8088a5\" (UID: \"fdbec8c6-4548-4f51-b683-2864ea8088a5\") "
Nov 01 07:09:31.369693 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.369512    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume" pod "fdbec8c6-4548-4f51-b683-2864ea8088a5" (UID: "fdbec8c6-4548-4f51-b683-2864ea8088a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:31.370031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.369996    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdbec8c6-4548-4f51-b683-2864ea8088a5-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:31.371172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.371149    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4" pod "fdbec8c6-4548-4f51-b683-2864ea8088a5" (UID: "fdbec8c6-4548-4f51-b683-2864ea8088a5"). InnerVolumeSpecName "kube-api-access-q4kf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:31.431457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.431420    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 07:09:31.470701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.470671    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume\") pod \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") "
Nov 01 07:09:31.470900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.470732    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28\" (UniqueName: \"kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28\") pod \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\" (UID: \"f297dfd2-be28-4b9d-8ec9-2566568b65c9\") "
Nov 01 07:09:31.471091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.471069    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume" pod "f297dfd2-be28-4b9d-8ec9-2566568b65c9" (UID: "f297dfd2-be28-4b9d-8ec9-2566568b65c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:31.471244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.471228    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4kf4\" (UniqueName: \"kubernetes.io/projected/fdbec8c6-4548-4f51-b683-2864ea8088a5-kube-api-access-q4kf4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:31.471303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.471250    4468 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f297dfd2-be28-4b9d-8ec9-2566568b65c9-plugins-config-volume\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:31.473133 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.473109    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28" pod "f297dfd2-be28-4b9d-8ec9-2566568b65c9" (UID: "f297dfd2-be28-4b9d-8ec9-2566568b65c9"). InnerVolumeSpecName "kube-api-access-6nj28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:31.572518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.572420    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nj28\" (UniqueName: \"kubernetes.io/projected/f297dfd2-be28-4b9d-8ec9-2566568b65c9-kube-api-access-6nj28\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:31.772862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.772824    4468 generic.go:356] "Generic (PLEG): container finished" podID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerID="e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9" exitCode=0
Nov 01 07:09:31.773075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.772874    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"
Nov 01 07:09:31.773075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.772910    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" event={"ID":"f297dfd2-be28-4b9d-8ec9-2566568b65c9","Type":"ContainerDied","Data":"e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9"}
Nov 01 07:09:31.773075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.772961    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq" event={"ID":"f297dfd2-be28-4b9d-8ec9-2566568b65c9","Type":"ContainerDied","Data":"246dd23587476fc8094820bf4a3ef573d923d58345c57e26103b9bb54e527d47"}
Nov 01 07:09:31.773075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.772992    4468 scope.go:122] "RemoveContainer" containerID="e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9"
Nov 01 07:09:31.774575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.774542    4468 generic.go:356] "Generic (PLEG): container finished" podID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerID="4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410" exitCode=0
Nov 01 07:09:31.774662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.774607    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"
Nov 01 07:09:31.774662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.774654    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" event={"ID":"fdbec8c6-4548-4f51-b683-2864ea8088a5","Type":"ContainerDied","Data":"4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410"}
Nov 01 07:09:31.774749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.774683    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb" event={"ID":"fdbec8c6-4548-4f51-b683-2864ea8088a5","Type":"ContainerDied","Data":"ecd6f4a41a290dfb933780e4c604d2734c476264875c0ca3646ecb3947b3a413"}
Nov 01 07:09:31.782936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.782915    4468 scope.go:122] "RemoveContainer" containerID="e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9"
Nov 01 07:09:31.783293 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:31.783274    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9\": container with ID starting with e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9 not found: ID does not exist" containerID="e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9"
Nov 01 07:09:31.783382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.783304    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9"} err="failed to get container status \"e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9\": rpc error: code = NotFound desc = could not find container \"e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9\": container with ID starting with e01bd6e79510a234f97a04274c5cdf5de85ab1e0e8ce11810fb2c2907dd0fbe9 not found: ID does not exist"
Nov 01 07:09:31.783382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.783334    4468 scope.go:122] "RemoveContainer" containerID="4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410"
Nov 01 07:09:31.791949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.791932    4468 scope.go:122] "RemoveContainer" containerID="4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410"
Nov 01 07:09:31.792201 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:31.792184    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410\": container with ID starting with 4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410 not found: ID does not exist" containerID="4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410"
Nov 01 07:09:31.792267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.792216    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410"} err="failed to get container status \"4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410\": rpc error: code = NotFound desc = could not find container \"4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410\": container with ID starting with 4154e02cf910ac6aeb4dab189851e9765cfbaec1af10e02074e3817631850410 not found: ID does not exist"
Nov 01 07:09:31.937097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.937007    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"]
Nov 01 07:09:31.944061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:31.944026    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-190724-8bb994975-5nxgq"]
Nov 01 07:09:32.046959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:32.046925    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"]
Nov 01 07:09:32.050769 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:32.050748    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-190724-65b8997649-4kvmb"]
Nov 01 07:09:32.592822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:32.592791    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" path="/var/lib/kubelet/pods/f297dfd2-be28-4b9d-8ec9-2566568b65c9/volumes"
Nov 01 07:09:32.593319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:32.593277    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" path="/var/lib/kubelet/pods/fdbec8c6-4548-4f51-b683-2864ea8088a5/volumes"
Nov 01 07:09:32.948062 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:32.947985    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:09:43.956263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:43.956230    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:09:48.173160 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:48.173121    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"]
Nov 01 07:09:48.173573 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:48.173321    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" containerID="cri-o://55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098" gracePeriod=30
Nov 01 07:09:48.188145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:48.188116    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"]
Nov 01 07:09:48.188334 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:48.188298    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" containerID="cri-o://f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67" gracePeriod=30
Nov 01 07:09:50.903308 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:50.903266    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:09:50.903739 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:50.903349    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:09:51.248293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:51.248218    4468 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:09:51.248293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:51.248276    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:09:52.241485 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:52.241434    4468 scope.go:122] "RemoveContainer" containerID="88498cb4145dbb1794fb4974a040a60941c2d538a68a88996c2d7271b6d8ec15"
Nov 01 07:09:52.252811 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:52.252784    4468 scope.go:122] "RemoveContainer" containerID="7a5c46c3b13a4191a88ae6f699b94f6cb757c3070625c8d4e74397f59886e43b"
Nov 01 07:09:52.264938 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:52.264909    4468 scope.go:122] "RemoveContainer" containerID="10e6bb87223ec84b1b2e44356619bf3183a6e438e555607e37ef0b9ffa67522f"
Nov 01 07:09:52.274164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:52.274143    4468 scope.go:122] "RemoveContainer" containerID="3fa1a9b0b3f244edcbf63e5368f9d51dd2d16b4f134a09c1ff49c8b2b5806aa3"
Nov 01 07:09:53.447068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.447002    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:09:53.469131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.469109    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:09:53.605386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605357    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.605560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605434    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.605560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605459    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.605560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605500    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.605560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605537    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.605780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605575    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.605780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605639    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.605780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605664    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.605780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605702    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.605780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605752    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.606058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605785    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.606058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605838    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.606058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605873    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.606058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.605927    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.606058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606001    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.606058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606041    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.606428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606060    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:53.606428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606118    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.606428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606155    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.606428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606190    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.606428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606246    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw\") pod \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\" (UID: \"55d6dd7f-8540-4d4c-9b98-c6261e63a848\") "
Nov 01 07:09:53.606756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606440    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.606756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606564    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.606756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606598    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert\") pod \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\" (UID: \"2d16f984-5a55-499f-ad7c-afc1e3d07f67\") "
Nov 01 07:09:53.606756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606569    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.606991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.606905    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.607056 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607040    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607119 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607061    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607119 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607076    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607119 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607090    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607119 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607103    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607119 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607116    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607131    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607145    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.607650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607164    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.607650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.607556    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:09:53.608391 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.608365    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:09:53.608828 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.608664    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:09:53.609330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.609305    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:53.609330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.609321    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:53.609447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.609342    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "kube-api-access-sjkx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:53.609447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.609346    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.609447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.609373    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy" pod "2d16f984-5a55-499f-ad7c-afc1e3d07f67" (UID: "2d16f984-5a55-499f-ad7c-afc1e3d07f67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:09:53.610080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.610063    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw" pod "55d6dd7f-8540-4d4c-9b98-c6261e63a848" (UID: "55d6dd7f-8540-4d4c-9b98-c6261e63a848"). InnerVolumeSpecName "kube-api-access-rxlbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707668    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707708    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707718    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707727    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707735    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707745    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d6dd7f-8540-4d4c-9b98-c6261e63a848-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.707753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707755    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxlbw\" (UniqueName: \"kubernetes.io/projected/55d6dd7f-8540-4d4c-9b98-c6261e63a848-kube-api-access-rxlbw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.708104 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707765    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d16f984-5a55-499f-ad7c-afc1e3d07f67-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.708104 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707774    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjkx5\" (UniqueName: \"kubernetes.io/projected/2d16f984-5a55-499f-ad7c-afc1e3d07f67-kube-api-access-sjkx5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.708104 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.707782    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d16f984-5a55-499f-ad7c-afc1e3d07f67-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:09:53.975572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.975477    4468 generic.go:356] "Generic (PLEG): container finished" podID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerID="55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098" exitCode=0
Nov 01 07:09:53.975572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.975544    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"
Nov 01 07:09:53.975841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.975628    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" event={"ID":"55d6dd7f-8540-4d4c-9b98-c6261e63a848","Type":"ContainerDied","Data":"55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098"}
Nov 01 07:09:53.975841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.975739    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg" event={"ID":"55d6dd7f-8540-4d4c-9b98-c6261e63a848","Type":"ContainerDied","Data":"022a559a11be7bfc8159e76aca98a203315d101a46b43dd8cd72604f173a9c89"}
Nov 01 07:09:53.975841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.975805    4468 scope.go:122] "RemoveContainer" containerID="55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098"
Nov 01 07:09:53.977437 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.977417    4468 generic.go:356] "Generic (PLEG): container finished" podID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerID="f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67" exitCode=0
Nov 01 07:09:53.977582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.977479    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"
Nov 01 07:09:53.977582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.977528    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" event={"ID":"2d16f984-5a55-499f-ad7c-afc1e3d07f67","Type":"ContainerDied","Data":"f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67"}
Nov 01 07:09:53.977665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.977581    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb" event={"ID":"2d16f984-5a55-499f-ad7c-afc1e3d07f67","Type":"ContainerDied","Data":"0e5c3ceccb34740ff15b21836f486348c8dbfe30743a6f69e47c7356a09fe07f"}
Nov 01 07:09:53.987572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.987527    4468 scope.go:122] "RemoveContainer" containerID="55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098"
Nov 01 07:09:53.987931 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:53.987882    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098\": container with ID starting with 55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098 not found: ID does not exist" containerID="55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098"
Nov 01 07:09:53.987993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.987942    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098"} err="failed to get container status \"55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098\": rpc error: code = NotFound desc = could not find container \"55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098\": container with ID starting with 55df4ca44d0a05ed06f178ade90ad7758af4c22d06b2229e8e25fbf738a56098 not found: ID does not exist"
Nov 01 07:09:53.987993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:53.987964    4468 scope.go:122] "RemoveContainer" containerID="f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67"
Nov 01 07:09:54.000074 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.000054    4468 scope.go:122] "RemoveContainer" containerID="f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67"
Nov 01 07:09:54.000361 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:09:54.000340    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67\": container with ID starting with f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67 not found: ID does not exist" containerID="f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67"
Nov 01 07:09:54.000430 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.000369    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67"} err="failed to get container status \"f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67\": rpc error: code = NotFound desc = could not find container \"f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67\": container with ID starting with f3f4693f08493b86c65912e6c807ed7757dbcb596042ae418d2b2fd81f3b9b67 not found: ID does not exist"
Nov 01 07:09:54.141055 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.141022    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"]
Nov 01 07:09:54.146378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.146345    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8crkg"]
Nov 01 07:09:54.253751 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.253670    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"]
Nov 01 07:09:54.256155 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.256135    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-8klrb"]
Nov 01 07:09:54.590515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.590483    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" path="/var/lib/kubelet/pods/2d16f984-5a55-499f-ad7c-afc1e3d07f67/volumes"
Nov 01 07:09:54.590943 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:09:54.590929    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" path="/var/lib/kubelet/pods/55d6dd7f-8540-4d4c-9b98-c6261e63a848/volumes"
Nov 01 07:10:02.946955 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:10:02.946910    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:10:32.948586 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:10:32.948484    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:11:02.948784 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:11:02.948740    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:11:22.044638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:22.044588    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"]
Nov 01 07:11:22.093908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:22.093876    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727308fb-lk8zc"]
Nov 01 07:11:22.589093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:22.589054    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7454fa9-3a84-46e0-ad1d-1a5695d46833" path="/var/lib/kubelet/pods/b7454fa9-3a84-46e0-ad1d-1a5695d46833/volumes"
Nov 01 07:11:26.046974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:26.046937    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"]
Nov 01 07:11:26.098266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:26.098229    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7217ca6-vjdpl"]
Nov 01 07:11:26.589738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:26.589702    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcca7e06-8987-4310-be1b-145ea8bc38ce" path="/var/lib/kubelet/pods/bcca7e06-8987-4310-be1b-145ea8bc38ce/volumes"
Nov 01 07:11:32.948631 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:11:32.948591    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:11:35.045330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:35.045279    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"]
Nov 01 07:11:35.094658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:35.094618    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626716ee-8jxnw"]
Nov 01 07:11:36.590576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:36.590531    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a002d4-d056-46d8-a899-b0a5ac168847" path="/var/lib/kubelet/pods/96a002d4-d056-46d8-a899-b0a5ac168847/volumes"
Nov 01 07:11:39.044370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:39.044333    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"]
Nov 01 07:11:39.094108 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:39.094065    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad347a4-wnl9c"]
Nov 01 07:11:40.589676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:40.589644    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffdae7b-aff3-4318-b8a3-a31b4cc28aed" path="/var/lib/kubelet/pods/dffdae7b-aff3-4318-b8a3-a31b4cc28aed/volumes"
Nov 01 07:11:50.045315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:50.045273    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"]
Nov 01 07:11:50.094950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:50.094910    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9d352f-hvpkj"]
Nov 01 07:11:50.589535 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:11:50.589499    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811a356d-69a1-481f-89c7-fcc7e8ae1070" path="/var/lib/kubelet/pods/811a356d-69a1-481f-89c7-fcc7e8ae1070/volumes"
Nov 01 07:12:02.947241 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:12:02.947154    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:12:05.044027 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:05.043985    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"]
Nov 01 07:12:05.093375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:05.093333    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27248075-xmlgm"]
Nov 01 07:12:06.589644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:06.589601    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ec8e21-c2dd-4697-83ea-d1e78bb2e99d" path="/var/lib/kubelet/pods/85ec8e21-c2dd-4697-83ea-d1e78bb2e99d/volumes"
Nov 01 07:12:20.044691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:20.044654    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"]
Nov 01 07:12:20.093963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:20.093926    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4625eefa-vmp7c"]
Nov 01 07:12:20.589648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:20.589614    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6523b8d-3f80-4283-8e57-2d8eac4528f8" path="/var/lib/kubelet/pods/b6523b8d-3f80-4283-8e57-2d8eac4528f8/volumes"
Nov 01 07:12:22.043712 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:22.043674    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"]
Nov 01 07:12:22.092904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:22.092867    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921aa85c-k5h7b"]
Nov 01 07:12:22.589395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:22.589362    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235d2991-3c36-4099-bdcc-8a40338de1fc" path="/var/lib/kubelet/pods/235d2991-3c36-4099-bdcc-8a40338de1fc/volumes"
Nov 01 07:12:32.949117 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:12:32.949076    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:12:41.043911 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:41.043873    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"]
Nov 01 07:12:41.093324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:41.093291    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82ba365-7jvq9"]
Nov 01 07:12:42.589626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:42.589594    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8c09ec-e965-4343-9adf-833560b106fd" path="/var/lib/kubelet/pods/6b8c09ec-e965-4343-9adf-833560b106fd/volumes"
Nov 01 07:12:52.364502 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:52.364453    4468 scope.go:122] "RemoveContainer" containerID="e7c5a072d5b6248300893b1a06e26a6d624d9c422c1b3f0106be1e0cb6030f0e"
Nov 01 07:12:52.374587 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:52.374555    4468 scope.go:122] "RemoveContainer" containerID="69ce757342d7072ac3c069a6f556eb90c69b5bbd2d94b3363427d8db9e117f2f"
Nov 01 07:12:52.386762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:52.386739    4468 scope.go:122] "RemoveContainer" containerID="6afe46ef81b6108fd630ba2746857511bc460f862a50317b06477eb75a054656"
Nov 01 07:12:52.395695 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:52.395673    4468 scope.go:122] "RemoveContainer" containerID="deafd2e6e36644e860f1b022ca078b17421fd8c0a75274a752fb7526b5c425b3"
Nov 01 07:12:52.404547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:12:52.404524    4468 scope.go:122] "RemoveContainer" containerID="0c30d042980f400bbd950225f4e7862ede007cf0b5d7f6db61d0f933eaacd54a"
Nov 01 07:13:02.947102 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:13:02.947065    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:13:03.746954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.746917    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"]
Nov 01 07:13:03.747403 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747388    4468 cpu_manager.go:414] "containerMap: removing container" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerName="payload-processing"
Nov 01 07:13:03.747454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747405    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerName="payload-processing"
Nov 01 07:13:03.747454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747425    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerName="payload-pre-processing"
Nov 01 07:13:03.747454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747432    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerName="payload-pre-processing"
Nov 01 07:13:03.747454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747449    4468 cpu_manager.go:414] "containerMap: removing container" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerName="payload-processing"
Nov 01 07:13:03.747454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747454    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerName="payload-processing"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747484    4468 cpu_manager.go:414] "containerMap: removing container" podUID="00a12e3c-6346-46ca-ad37-55dac1c345c6" containerName="revoke-keys"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747491    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a12e3c-6346-46ca-ad37-55dac1c345c6" containerName="revoke-keys"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747499    4468 cpu_manager.go:414] "containerMap: removing container" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747504    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747512    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747517    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747603    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a12e3c-6346-46ca-ad37-55dac1c345c6" containerName="revoke-keys"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747612    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d6dd7f-8540-4d4c-9b98-c6261e63a848" containerName="istio-proxy"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747617    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d16f984-5a55-499f-ad7c-afc1e3d07f67" containerName="istio-proxy"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747629    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbec8c6-4548-4f51-b683-2864ea8088a5" containerName="payload-processing"
Nov 01 07:13:03.747640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747639    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f297dfd2-be28-4b9d-8ec9-2566568b65c9" containerName="payload-pre-processing"
Nov 01 07:13:03.747975 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.747649    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf29ea2f-d691-4df5-af90-4ff4cd63ba33" containerName="payload-processing"
Nov 01 07:13:03.750900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.750880    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.803082 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.803049    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:13:03.803327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.803307    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:13:03.852855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.852821    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:13:03.852855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.852833    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:13:03.855576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.855549    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"]
Nov 01 07:13:03.893965 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.893927    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.894128 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.894065    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvzv\" (UniqueName: \"kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.894176 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.894128    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.995477 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.995428    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.995872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.995543    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvzv\" (UniqueName: \"kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.995872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.995595    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.996286 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.996255    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:03.998024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:03.997977    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:04.051893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:04.051860    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvzv\" (UniqueName: \"kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv\") pod \"maas-api-revoke-sub-510896cc-trgkk\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:04.063716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:04.063692    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:04.434288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:04.434266    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"]
Nov 01 07:13:04.436479 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:13:04.436435    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a946347_3726_4a25_be00_53de646d2b94.slice/crio-1aafaa5c0b980cf90884813920ca89f9717ab2a4d6d8d1427472f33a0f3a29ee WatchSource:0}: Error finding container 1aafaa5c0b980cf90884813920ca89f9717ab2a4d6d8d1427472f33a0f3a29ee: Status 404 returned error can't find the container with id 1aafaa5c0b980cf90884813920ca89f9717ab2a4d6d8d1427472f33a0f3a29ee
Nov 01 07:13:05.118999 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:05.118958    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk" event={"ID":"9a946347-3726-4a25-be00-53de646d2b94","Type":"ContainerStarted","Data":"bf90c5160ca4c15d12c91bca207899d4394189751d6e7a44090f9e8350013cb8"}
Nov 01 07:13:05.119519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:05.119017    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk" event={"ID":"9a946347-3726-4a25-be00-53de646d2b94","Type":"ContainerStarted","Data":"1aafaa5c0b980cf90884813920ca89f9717ab2a4d6d8d1427472f33a0f3a29ee"}
Nov 01 07:13:05.231356 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:05.231296    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk" podStartSLOduration=2.231280954 podStartE2EDuration="2.231280954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:13:05.230533103 +0000 UTC m=+3796.798339934" watchObservedRunningTime="2026-11-01 07:13:05.231280954 +0000 UTC m=+3796.799087785"
Nov 01 07:13:06.125847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:06.125812    4468 generic.go:356] "Generic (PLEG): container finished" podID="9a946347-3726-4a25-be00-53de646d2b94" containerID="bf90c5160ca4c15d12c91bca207899d4394189751d6e7a44090f9e8350013cb8" exitCode=0
Nov 01 07:13:06.126311 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:06.125896    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk" event={"ID":"9a946347-3726-4a25-be00-53de646d2b94","Type":"ContainerDied","Data":"bf90c5160ca4c15d12c91bca207899d4394189751d6e7a44090f9e8350013cb8"}
Nov 01 07:13:07.269243 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.269217    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:07.328288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.328246    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv\" (UniqueName: \"kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv\") pod \"9a946347-3726-4a25-be00-53de646d2b94\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") "
Nov 01 07:13:07.328504 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.328307    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca\") pod \"9a946347-3726-4a25-be00-53de646d2b94\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") "
Nov 01 07:13:07.328504 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.328328    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca\") pod \"9a946347-3726-4a25-be00-53de646d2b94\" (UID: \"9a946347-3726-4a25-be00-53de646d2b94\") "
Nov 01 07:13:07.328686 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.328664    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca" pod "9a946347-3726-4a25-be00-53de646d2b94" (UID: "9a946347-3726-4a25-be00-53de646d2b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:13:07.330595 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.330554    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv" pod "9a946347-3726-4a25-be00-53de646d2b94" (UID: "9a946347-3726-4a25-be00-53de646d2b94"). InnerVolumeSpecName "kube-api-access-vhvzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:13:07.330595 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.330584    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca" pod "9a946347-3726-4a25-be00-53de646d2b94" (UID: "9a946347-3726-4a25-be00-53de646d2b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:13:07.429820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.429745    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhvzv\" (UniqueName: \"kubernetes.io/projected/9a946347-3726-4a25-be00-53de646d2b94-kube-api-access-vhvzv\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:07.429820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.429771    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a946347-3726-4a25-be00-53de646d2b94-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:07.429820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:07.429782    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a946347-3726-4a25-be00-53de646d2b94-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:08.049803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:08.049765    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"]
Nov 01 07:13:08.098648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:08.098609    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"]
Nov 01 07:13:08.138701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:08.138665    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk" event={"ID":"9a946347-3726-4a25-be00-53de646d2b94","Type":"ContainerDied","Data":"1aafaa5c0b980cf90884813920ca89f9717ab2a4d6d8d1427472f33a0f3a29ee"}
Nov 01 07:13:08.138701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:08.138683    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"
Nov 01 07:13:08.138901 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:08.138718    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aafaa5c0b980cf90884813920ca89f9717ab2a4d6d8d1427472f33a0f3a29ee"
Nov 01 07:13:12.206314 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:12.206283    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:13:15.887366 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.887334    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"]
Nov 01 07:13:15.887876 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.887858    4468 cpu_manager.go:414] "containerMap: removing container" podUID="9a946347-3726-4a25-be00-53de646d2b94" containerName="revoke-keys"
Nov 01 07:13:15.887946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.887878    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a946347-3726-4a25-be00-53de646d2b94" containerName="revoke-keys"
Nov 01 07:13:15.888006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.887992    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a946347-3726-4a25-be00-53de646d2b94" containerName="revoke-keys"
Nov 01 07:13:15.891422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.891406    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:15.909800 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.909771    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:15.909949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.909926    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klt4b\" (UniqueName: \"kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:15.910006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.909982    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:15.944512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.944456    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:13:15.944808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.944644    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:13:15.994602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.994556    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:13:15.994602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.994592    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:13:15.997118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:15.997092    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"]
Nov 01 07:13:16.011552 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.011519    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klt4b\" (UniqueName: \"kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.011710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.011599    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.011710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.011673    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.012301 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.012271    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.014636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.014608    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.068001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.067969    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klt4b\" (UniqueName: \"kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b\") pod \"maas-api-revoke-sub-8ec89464-6x9gn\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.204895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.204820    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:16.582263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:16.582241    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"]
Nov 01 07:13:16.584333 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:13:16.584287    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f881bc_4c71_4442_bb45_d7bff48c91f4.slice/crio-8016213bfb09d8c99166dc21dc12e02e323d4ad4f0362aeb18b4f8ed276fd5ed WatchSource:0}: Error finding container 8016213bfb09d8c99166dc21dc12e02e323d4ad4f0362aeb18b4f8ed276fd5ed: Status 404 returned error can't find the container with id 8016213bfb09d8c99166dc21dc12e02e323d4ad4f0362aeb18b4f8ed276fd5ed
Nov 01 07:13:17.192391 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:17.192292    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn" event={"ID":"d9f881bc-4c71-4442-bb45-d7bff48c91f4","Type":"ContainerStarted","Data":"6ba6df317fbf966da7bc49949ba7a72de9571039000c9018fa68c502c8e256ed"}
Nov 01 07:13:17.192391 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:17.192352    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn" event={"ID":"d9f881bc-4c71-4442-bb45-d7bff48c91f4","Type":"ContainerStarted","Data":"8016213bfb09d8c99166dc21dc12e02e323d4ad4f0362aeb18b4f8ed276fd5ed"}
Nov 01 07:13:17.304071 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:17.304008    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn" podStartSLOduration=2.303992536 podStartE2EDuration="2.303992536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:13:17.303065075 +0000 UTC m=+3808.870871906" watchObservedRunningTime="2026-11-01 07:13:17.303992536 +0000 UTC m=+3808.871799365"
Nov 01 07:13:18.199627 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:18.199590    4468 generic.go:356] "Generic (PLEG): container finished" podID="d9f881bc-4c71-4442-bb45-d7bff48c91f4" containerID="6ba6df317fbf966da7bc49949ba7a72de9571039000c9018fa68c502c8e256ed" exitCode=0
Nov 01 07:13:18.200096 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:18.199682    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn" event={"ID":"d9f881bc-4c71-4442-bb45-d7bff48c91f4","Type":"ContainerDied","Data":"6ba6df317fbf966da7bc49949ba7a72de9571039000c9018fa68c502c8e256ed"}
Nov 01 07:13:19.087962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.087926    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"]
Nov 01 07:13:19.102799 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.102764    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4206727-jppvd"]
Nov 01 07:13:19.342978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.342921    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:19.450978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.450947    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b\" (UniqueName: \"kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b\") pod \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") "
Nov 01 07:13:19.451164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.451099    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca\") pod \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") "
Nov 01 07:13:19.451164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.451134    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca\") pod \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\" (UID: \"d9f881bc-4c71-4442-bb45-d7bff48c91f4\") "
Nov 01 07:13:19.451523 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.451499    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca" pod "d9f881bc-4c71-4442-bb45-d7bff48c91f4" (UID: "d9f881bc-4c71-4442-bb45-d7bff48c91f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:13:19.453245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.453222    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b" pod "d9f881bc-4c71-4442-bb45-d7bff48c91f4" (UID: "d9f881bc-4c71-4442-bb45-d7bff48c91f4"). InnerVolumeSpecName "kube-api-access-klt4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:13:19.453335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.453288    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca" pod "d9f881bc-4c71-4442-bb45-d7bff48c91f4" (UID: "d9f881bc-4c71-4442-bb45-d7bff48c91f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:13:19.552375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.552343    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:19.552375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.552371    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f881bc-4c71-4442-bb45-d7bff48c91f4-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:19.552375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:19.552383    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klt4b\" (UniqueName: \"kubernetes.io/projected/d9f881bc-4c71-4442-bb45-d7bff48c91f4-kube-api-access-klt4b\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:20.212458 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:20.212428    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"
Nov 01 07:13:20.212458 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:20.212478    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn" event={"ID":"d9f881bc-4c71-4442-bb45-d7bff48c91f4","Type":"ContainerDied","Data":"8016213bfb09d8c99166dc21dc12e02e323d4ad4f0362aeb18b4f8ed276fd5ed"}
Nov 01 07:13:20.212758 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:20.212528    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8016213bfb09d8c99166dc21dc12e02e323d4ad4f0362aeb18b4f8ed276fd5ed"
Nov 01 07:13:20.589273 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:20.589230    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bf930b-43d5-4357-8ff2-826c5940b15f" path="/var/lib/kubelet/pods/43bf930b-43d5-4357-8ff2-826c5940b15f/volumes"
Nov 01 07:13:23.070849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.070802    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks" podUID="34dd1ad4-aed2-42bd-8d09-a01312d1fb84" containerName="main" containerID="cri-o://ad1ea9ded7ef036724efd22c92a834007cb8978ad63e4d55c6d8bcd78b8a6f93" gracePeriod=45
Nov 01 07:13:23.120450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.120403    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" podUID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" containerName="main" containerID="cri-o://35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99" gracePeriod=45
Nov 01 07:13:23.133189 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.133156    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" podUID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": dial tcp 10.133.0.165:8000: connect: connection refused"
Nov 01 07:13:23.232058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.232014    4468 generic.go:356] "Generic (PLEG): container finished" podID="34dd1ad4-aed2-42bd-8d09-a01312d1fb84" containerID="ad1ea9ded7ef036724efd22c92a834007cb8978ad63e4d55c6d8bcd78b8a6f93" exitCode=0
Nov 01 07:13:23.232184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.232084    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks" event={"ID":"34dd1ad4-aed2-42bd-8d09-a01312d1fb84","Type":"ContainerDied","Data":"ad1ea9ded7ef036724efd22c92a834007cb8978ad63e4d55c6d8bcd78b8a6f93"}
Nov 01 07:13:23.384107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.384075    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 07:13:23.406015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.405986    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 07:13:23.499770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.499736    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs\" (UniqueName: \"kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs\") pod \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") "
Nov 01 07:13:23.499770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.499779    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm\") pod \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") "
Nov 01 07:13:23.500045 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.499809    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home\") pod \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") "
Nov 01 07:13:23.500045 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.499945    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home\") pod \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") "
Nov 01 07:13:23.500045 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500002    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir\") pod \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") "
Nov 01 07:13:23.500045 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500037    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs\" (UniqueName: \"kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs\") pod \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") "
Nov 01 07:13:23.500281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500075    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home" pod "34dd1ad4-aed2-42bd-8d09-a01312d1fb84" (UID: "34dd1ad4-aed2-42bd-8d09-a01312d1fb84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.500281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500081    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8\" (UniqueName: \"kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8\") pod \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") "
Nov 01 07:13:23.500281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500122    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache\") pod \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") "
Nov 01 07:13:23.500281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500138    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home" pod "67527b2f-0dc7-4e36-ab04-6ccd371d2dff" (UID: "67527b2f-0dc7-4e36-ab04-6ccd371d2dff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.500281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500185    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm\") pod \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\" (UID: \"34dd1ad4-aed2-42bd-8d09-a01312d1fb84\") "
Nov 01 07:13:23.500281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500260    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz\" (UniqueName: \"kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz\") pod \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") "
Nov 01 07:13:23.500792 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500301    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir\") pod \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") "
Nov 01 07:13:23.500792 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500361    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache\") pod \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\" (UID: \"67527b2f-0dc7-4e36-ab04-6ccd371d2dff\") "
Nov 01 07:13:23.500792 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500710    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir" pod "34dd1ad4-aed2-42bd-8d09-a01312d1fb84" (UID: "34dd1ad4-aed2-42bd-8d09-a01312d1fb84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.500966 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.500916    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache" pod "67527b2f-0dc7-4e36-ab04-6ccd371d2dff" (UID: "67527b2f-0dc7-4e36-ab04-6ccd371d2dff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.501562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.501091    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.501562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.501115    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.501562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.501128    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.501562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.501141    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.501562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.501228    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache" pod "34dd1ad4-aed2-42bd-8d09-a01312d1fb84" (UID: "34dd1ad4-aed2-42bd-8d09-a01312d1fb84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.501562 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.501399    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir" pod "67527b2f-0dc7-4e36-ab04-6ccd371d2dff" (UID: "67527b2f-0dc7-4e36-ab04-6ccd371d2dff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.504223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.503744    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm" pod "67527b2f-0dc7-4e36-ab04-6ccd371d2dff" (UID: "67527b2f-0dc7-4e36-ab04-6ccd371d2dff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.504223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.503897    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz" pod "67527b2f-0dc7-4e36-ab04-6ccd371d2dff" (UID: "67527b2f-0dc7-4e36-ab04-6ccd371d2dff"). InnerVolumeSpecName "kube-api-access-sglvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:13:23.504223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.504136    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8" pod "34dd1ad4-aed2-42bd-8d09-a01312d1fb84" (UID: "34dd1ad4-aed2-42bd-8d09-a01312d1fb84"). InnerVolumeSpecName "kube-api-access-7kzd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:13:23.504419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.504277    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm" pod "34dd1ad4-aed2-42bd-8d09-a01312d1fb84" (UID: "34dd1ad4-aed2-42bd-8d09-a01312d1fb84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:13:23.505626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.505603    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs" pod "67527b2f-0dc7-4e36-ab04-6ccd371d2dff" (UID: "67527b2f-0dc7-4e36-ab04-6ccd371d2dff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:13:23.505720 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.505622    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs" pod "34dd1ad4-aed2-42bd-8d09-a01312d1fb84" (UID: "34dd1ad4-aed2-42bd-8d09-a01312d1fb84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602691    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602727    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzd8\" (UniqueName: \"kubernetes.io/projected/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-kube-api-access-7kzd8\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602738    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602749    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602758    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sglvz\" (UniqueName: \"kubernetes.io/projected/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-kube-api-access-sglvz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602766    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602774    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dd1ad4-aed2-42bd-8d09-a01312d1fb84-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:23.602787 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:23.602782    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67527b2f-0dc7-4e36-ab04-6ccd371d2dff-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:13:24.239395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.239358    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"
Nov 01 07:13:24.239395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.239374    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks" event={"ID":"34dd1ad4-aed2-42bd-8d09-a01312d1fb84","Type":"ContainerDied","Data":"e7551ea7319bf043e25296bdf68a61907f55b7e70d81231733a5fa994d921c05"}
Nov 01 07:13:24.239964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.239424    4468 scope.go:122] "RemoveContainer" containerID="ad1ea9ded7ef036724efd22c92a834007cb8978ad63e4d55c6d8bcd78b8a6f93"
Nov 01 07:13:24.240900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.240828    4468 generic.go:356] "Generic (PLEG): container finished" podID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" containerID="35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99" exitCode=0
Nov 01 07:13:24.240900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.240879    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"
Nov 01 07:13:24.241017 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.240916    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" event={"ID":"67527b2f-0dc7-4e36-ab04-6ccd371d2dff","Type":"ContainerDied","Data":"35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99"}
Nov 01 07:13:24.241017 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.240956    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np" event={"ID":"67527b2f-0dc7-4e36-ab04-6ccd371d2dff","Type":"ContainerDied","Data":"342a67955825092847b2cf46d3b8070175edf3f364daa78f9d9c491b564d6a0e"}
Nov 01 07:13:24.249944 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.249922    4468 scope.go:122] "RemoveContainer" containerID="35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99"
Nov 01 07:13:24.258936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.258912    4468 scope.go:122] "RemoveContainer" containerID="35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99"
Nov 01 07:13:24.259259 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:13:24.259238    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99\": container with ID starting with 35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99 not found: ID does not exist" containerID="35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99"
Nov 01 07:13:24.259315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.259270    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99"} err="failed to get container status \"35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99\": rpc error: code = NotFound desc = could not find container \"35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99\": container with ID starting with 35a828b366d65cc319aa0e6c6b8e04051c847ed95b62a367b38d8bbbc1626a99 not found: ID does not exist"
Nov 01 07:13:24.402596 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.402497    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"]
Nov 01 07:13:24.452230 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.452197    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/simulator-w3-28c598-kserve-5cfc5c4479-kd9np"]
Nov 01 07:13:24.559129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.559092    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"]
Nov 01 07:13:24.562611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.562585    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-28c598/premium-w3-28c598-kserve-58bbf6b898-7blks"]
Nov 01 07:13:24.589545 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.589510    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34dd1ad4-aed2-42bd-8d09-a01312d1fb84" path="/var/lib/kubelet/pods/34dd1ad4-aed2-42bd-8d09-a01312d1fb84/volumes"
Nov 01 07:13:24.589951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:24.589938    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" path="/var/lib/kubelet/pods/67527b2f-0dc7-4e36-ab04-6ccd371d2dff/volumes"
Nov 01 07:13:30.391063 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:30.390973    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:13:31.044408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:31.044377    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"]
Nov 01 07:13:31.093638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:31.093611    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24abdc7e-42fnr"]
Nov 01 07:13:32.590712 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:32.590681    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e98e889-416e-4981-8b2a-bc077c650bc6" path="/var/lib/kubelet/pods/9e98e889-416e-4981-8b2a-bc077c650bc6/volumes"
Nov 01 07:13:32.946987 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:13:32.946898    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:13:43.440855 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:43.440816    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:13:52.442208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:52.442177    4468 scope.go:122] "RemoveContainer" containerID="eb8c34c27c1be30bf37818022e757000c8bddd8ee0df3efff7d055c7f6816e7e"
Nov 01 07:13:52.451617 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:52.451598    4468 scope.go:122] "RemoveContainer" containerID="51d783c519e13d4cad3b75ad45de911b01d6a4208bbf62c8e30311a933051897"
Nov 01 07:13:52.459658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:52.459636    4468 scope.go:122] "RemoveContainer" containerID="db48bd4b909ee3f7448dd0bb5b6f19bf0a3be7d68a3506d683816e71e7a46e8f"
Nov 01 07:13:52.467932 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:52.467906    4468 scope.go:122] "RemoveContainer" containerID="af3a8635e5f185eeee5c0a4e28cf742c7df32e938abd3dfeed043d64c19f68da"
Nov 01 07:13:54.044240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:54.044205    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"]
Nov 01 07:13:54.093186 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:54.093153    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63fb107-dh9pq"]
Nov 01 07:13:54.592382 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:54.592333    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2" path="/var/lib/kubelet/pods/ff2ceea1-f3e6-46a4-a7f9-e0fd3b7dd7e2/volumes"
Nov 01 07:13:55.045353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:55.045306    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"]
Nov 01 07:13:55.094533 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:55.094501    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f689c3d-xslfj"]
Nov 01 07:13:56.589312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:13:56.589281    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744cd94b-5f35-4b48-8caf-905b8f17890c" path="/var/lib/kubelet/pods/744cd94b-5f35-4b48-8caf-905b8f17890c/volumes"
Nov 01 07:14:02.947265 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:14:02.947227    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:14:05.291110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:05.291074    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:14:06.049542 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:06.049508    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"]
Nov 01 07:14:06.051910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:06.051887    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae3943-krfqr"]
Nov 01 07:14:06.590976 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:06.590935    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e882d15-b551-44cc-b76d-74cf56600887" path="/var/lib/kubelet/pods/3e882d15-b551-44cc-b76d-74cf56600887/volumes"
Nov 01 07:14:08.358669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:08.358617    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"]
Nov 01 07:14:08.359131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:08.358844    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy" containerID="cri-o://86f084dd07c92977d5d6b9c98fa4764298028432831a574c83f1e0a58292b554" gracePeriod=30
Nov 01 07:14:09.046984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:09.046950    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"]
Nov 01 07:14:09.048860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:09.048840    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3fce63-86sn7"]
Nov 01 07:14:10.590208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:10.590169    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f" path="/var/lib/kubelet/pods/80bcbb8e-44a4-4fb8-b0a8-3a61c72b7a7f/volumes"
Nov 01 07:14:13.522491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.522431    4468 generic.go:356] "Generic (PLEG): container finished" podID="e262958f-fc5e-4b5d-b231-775478f412b8" containerID="86f084dd07c92977d5d6b9c98fa4764298028432831a574c83f1e0a58292b554" exitCode=0
Nov 01 07:14:13.522929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.522508    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" event={"ID":"e262958f-fc5e-4b5d-b231-775478f412b8","Type":"ContainerDied","Data":"86f084dd07c92977d5d6b9c98fa4764298028432831a574c83f1e0a58292b554"}
Nov 01 07:14:13.522929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.522594    4468 scope.go:122] "RemoveContainer" containerID="15b110063d99ec22033317a5ba213972b2a5ae5b1d40e0279d9930cd38493cf8"
Nov 01 07:14:13.624706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.624632    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 07:14:13.636206 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636184    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636251    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636271    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636298    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636319    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636390    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636438    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636546    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636600    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert\") pod \"e262958f-fc5e-4b5d-b231-775478f412b8\" (UID: \"e262958f-fc5e-4b5d-b231-775478f412b8\") "
Nov 01 07:14:13.636759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636646    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:13.636759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636663    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:13.636918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636762    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:13.636918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.636882    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:13.637245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.637222    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:14:13.637374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.637355    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.637482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.637378    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.637482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.637393    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.637482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.637407    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.637482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.637419    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e262958f-fc5e-4b5d-b231-775478f412b8-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.638771 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.638716    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:13.638893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.638863    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:13.638893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.638866    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:14:13.638982 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.638893    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt" pod "e262958f-fc5e-4b5d-b231-775478f412b8" (UID: "e262958f-fc5e-4b5d-b231-775478f412b8"). InnerVolumeSpecName "kube-api-access-jpxwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:13.738193 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.738161    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e262958f-fc5e-4b5d-b231-775478f412b8-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.738193 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.738188    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.738408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.738201    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxwt\" (UniqueName: \"kubernetes.io/projected/e262958f-fc5e-4b5d-b231-775478f412b8-kube-api-access-jpxwt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:13.738408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:13.738211    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e262958f-fc5e-4b5d-b231-775478f412b8-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:14.530002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:14.529967    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp" event={"ID":"e262958f-fc5e-4b5d-b231-775478f412b8","Type":"ContainerDied","Data":"1f7e9fc7232dd10de0c24d4a26e7ae8ca72d43bbcad652fda85f8d384da5f77c"}
Nov 01 07:14:14.530420 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:14.530011    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"
Nov 01 07:14:14.530420 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:14.530054    4468 scope.go:122] "RemoveContainer" containerID="86f084dd07c92977d5d6b9c98fa4764298028432831a574c83f1e0a58292b554"
Nov 01 07:14:14.742908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:14.742871    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"]
Nov 01 07:14:14.792221 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:14.792137    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e0116-openshift-default-65b758dfff-59hsp"]
Nov 01 07:14:16.588698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:16.588666    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" path="/var/lib/kubelet/pods/e262958f-fc5e-4b5d-b231-775478f412b8/volumes"
Nov 01 07:14:22.067075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:22.067038    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"]
Nov 01 07:14:22.117500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:22.117433    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e416163-vpsxp"]
Nov 01 07:14:22.597057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:22.596999    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a12e3c-6346-46ca-ad37-55dac1c345c6" path="/var/lib/kubelet/pods/00a12e3c-6346-46ca-ad37-55dac1c345c6/volumes"
Nov 01 07:14:26.200972 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:26.200943    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:14:28.397561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:28.397522    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"]
Nov 01 07:14:28.397992 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:28.397779    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" containerID="cri-o://c35d4acbd12322a22d62e94142f995a38ecae63163ed2bbc4b64f2a95da34237" gracePeriod=30
Nov 01 07:14:28.600079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:28.600044    4468 patch_prober.go:29] interesting pod/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 07:14:28.600275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:28.600118    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 07:14:31.694577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:31.694544    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-b57dc_81cbf762-f2c8-44ae-b2a5-e1a8e7166be8/limitador/0.log"
Nov 01 07:14:32.948085 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:14:32.948047    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:14:33.548758 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.548694    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pwf6c_409da98c-561b-4be9-99cc-6535a33b667f/manager/0.log"
Nov 01 07:14:33.634797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.634770    4468 generic.go:356] "Generic (PLEG): container finished" podID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerID="c35d4acbd12322a22d62e94142f995a38ecae63163ed2bbc4b64f2a95da34237" exitCode=0
Nov 01 07:14:33.634917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.634838    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" event={"ID":"1ef0a7a7-9749-464c-8726-74d5b5210eb7","Type":"ContainerDied","Data":"c35d4acbd12322a22d62e94142f995a38ecae63163ed2bbc4b64f2a95da34237"}
Nov 01 07:14:33.634917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.634898    4468 scope.go:122] "RemoveContainer" containerID="8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380"
Nov 01 07:14:33.655568 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:14:33.655539    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380\": container with ID starting with 8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380 not found: ID does not exist" containerID="8803b06c59c97b487f765f89c7774a9755fe5203b6715957416f9f7419819380"
Nov 01 07:14:33.658883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.658866    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 07:14:33.750797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.750710    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.750797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.750790    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.750948    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751033    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:33.751089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751040    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751075    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751141    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751188    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751231    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751287    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket\") pod \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\" (UID: \"1ef0a7a7-9749-464c-8726-74d5b5210eb7\") "
Nov 01 07:14:33.751425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751368    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:14:33.751579 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751446    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:33.751694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751655    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:33.751885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.751862    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:33.752028 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.752010    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.752102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.752035    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.752102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.752049    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.752102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.752064    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.753243 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.753214    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "kube-api-access-hhwg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:33.753442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.753417    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:14:33.753825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.753804    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:14:33.753912 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.753839    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token" pod "1ef0a7a7-9749-464c-8726-74d5b5210eb7" (UID: "1ef0a7a7-9749-464c-8726-74d5b5210eb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:33.852827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.852792    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwg4\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-kube-api-access-hhwg4\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.852827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.852824    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.852827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.852834    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.853088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.852843    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef0a7a7-9749-464c-8726-74d5b5210eb7-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:33.853088 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:33.852852    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef0a7a7-9749-464c-8726-74d5b5210eb7-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:34.641701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:34.641672    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"
Nov 01 07:14:34.642172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:34.641671    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss" event={"ID":"1ef0a7a7-9749-464c-8726-74d5b5210eb7","Type":"ContainerDied","Data":"95132dbe05dfe6837234e9c20e73393787cfef8118cef7cd0f43be72c82c8114"}
Nov 01 07:14:34.642172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:34.641798    4468 scope.go:122] "RemoveContainer" containerID="c35d4acbd12322a22d62e94142f995a38ecae63163ed2bbc4b64f2a95da34237"
Nov 01 07:14:34.721459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:34.721416    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d475bcc9-z7whx_3294a44f-c41c-46fa-9844-842944e0ad5e/manager/1.log"
Nov 01 07:14:34.854089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:34.854002    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"]
Nov 01 07:14:34.903577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:34.903543    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-51aa8f-openshift-default-7f4595c85f-zqvss"]
Nov 01 07:14:35.488862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:35.488828    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xg8kb_f3ec2313-e247-4d2a-8d15-465ba35bd9f3/proxy/0.log"
Nov 01 07:14:36.129485 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:36.129443    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d475bcc9-z7whx_3294a44f-c41c-46fa-9844-842944e0ad5e/manager/0.log"
Nov 01 07:14:36.589598 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:36.589562    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" path="/var/lib/kubelet/pods/1ef0a7a7-9749-464c-8726-74d5b5210eb7/volumes"
Nov 01 07:14:52.507013 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:52.506983    4468 scope.go:122] "RemoveContainer" containerID="12c14e4d18a84fbd2c7557f3f0ac620d48764945fbb1391ff02887821bda0f3f"
Nov 01 07:14:52.516421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:52.516399    4468 scope.go:122] "RemoveContainer" containerID="4a1c972fd533981f10d07809de0e7c7edb911d1941a44bbc02258a042e2f4766"
Nov 01 07:14:52.524588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:52.524569    4468 scope.go:122] "RemoveContainer" containerID="7cf5a48d502a77c7d1a55450699297473733b401caa8b381e7447616a02121f6"
Nov 01 07:14:52.532666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:52.532646    4468 scope.go:122] "RemoveContainer" containerID="27e7a0966916a932e01749bb7a4ff1928f64096bb0c15f0bc8e9884d4d948e21"
Nov 01 07:14:59.612406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:14:59.612325    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:15:00.158231 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158198    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"]
Nov 01 07:15:00.158669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158654    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158670    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158688    4468 cpu_manager.go:414] "containerMap: removing container" podUID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" containerName="main"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158693    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" containerName="main"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158700    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d9f881bc-4c71-4442-bb45-d7bff48c91f4" containerName="revoke-keys"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158705    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f881bc-4c71-4442-bb45-d7bff48c91f4" containerName="revoke-keys"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158726    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158731    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158743    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158747    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158754    4468 cpu_manager.go:414] "containerMap: removing container" podUID="34dd1ad4-aed2-42bd-8d09-a01312d1fb84" containerName="main"
Nov 01 07:15:00.158759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158758    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34dd1ad4-aed2-42bd-8d09-a01312d1fb84" containerName="main"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158822    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158835    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="34dd1ad4-aed2-42bd-8d09-a01312d1fb84" containerName="main"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158842    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158850    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158857    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="67527b2f-0dc7-4e36-ab04-6ccd371d2dff" containerName="main"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158865    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f881bc-4c71-4442-bb45-d7bff48c91f4" containerName="revoke-keys"
Nov 01 07:15:00.159321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.158874    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef0a7a7-9749-464c-8726-74d5b5210eb7" containerName="istio-proxy"
Nov 01 07:15:00.161917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.161902    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.222766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.222711    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqsm\" (UniqueName: \"kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm\") pod \"maas-api-key-cleanup-29891955-jmdl8\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.222980 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.222857    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891955-jmdl8\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.265615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.265566    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:15:00.265615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.265597    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:15:00.265862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.265650    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:15:00.267869 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.267848    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"]
Nov 01 07:15:00.323689 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.323658    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqsm\" (UniqueName: \"kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm\") pod \"maas-api-key-cleanup-29891955-jmdl8\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.323906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.323710    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891955-jmdl8\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.324523 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.324495    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891955-jmdl8\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.380567 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.380532    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqsm\" (UniqueName: \"kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm\") pod \"maas-api-key-cleanup-29891955-jmdl8\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.475308 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.475225    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:00.849415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:00.849387    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"]
Nov 01 07:15:00.851537 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:15:00.851508    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad1437a_ac0a_4501_a3ee_1cb870cd81f9.slice/crio-e8e7d0e097308810da1221bbd3cc06d43a3f26f2e1f3adc44ff74dcf90b09a54 WatchSource:0}: Error finding container e8e7d0e097308810da1221bbd3cc06d43a3f26f2e1f3adc44ff74dcf90b09a54: Status 404 returned error can't find the container with id e8e7d0e097308810da1221bbd3cc06d43a3f26f2e1f3adc44ff74dcf90b09a54
Nov 01 07:15:01.794973 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:01.794931    4468 generic.go:356] "Generic (PLEG): container finished" podID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerID="1b9021fc90cf61400d0b74c6c2bca3054ee4e721f8f0c5589be503ecb3ced486" exitCode=60
Nov 01 07:15:01.795171 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:01.795020    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" event={"ID":"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9","Type":"ContainerDied","Data":"1b9021fc90cf61400d0b74c6c2bca3054ee4e721f8f0c5589be503ecb3ced486"}
Nov 01 07:15:01.795171 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:01.795079    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" event={"ID":"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9","Type":"ContainerStarted","Data":"e8e7d0e097308810da1221bbd3cc06d43a3f26f2e1f3adc44ff74dcf90b09a54"}
Nov 01 07:15:01.795272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:01.795245    4468 scope.go:122] "RemoveContainer" containerID="1b9021fc90cf61400d0b74c6c2bca3054ee4e721f8f0c5589be503ecb3ced486"
Nov 01 07:15:02.802356 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:02.802314    4468 generic.go:356] "Generic (PLEG): container finished" podID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerID="59d239291b17afe9c4d98b8945243416a495bc4edc4299fa3c27d010dcada6e2" exitCode=60
Nov 01 07:15:02.802900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:02.802410    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" event={"ID":"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9","Type":"ContainerDied","Data":"59d239291b17afe9c4d98b8945243416a495bc4edc4299fa3c27d010dcada6e2"}
Nov 01 07:15:02.802900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:02.802504    4468 scope.go:122] "RemoveContainer" containerID="1b9021fc90cf61400d0b74c6c2bca3054ee4e721f8f0c5589be503ecb3ced486"
Nov 01 07:15:02.802900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:02.802780    4468 scope.go:122] "RemoveContainer" containerID="59d239291b17afe9c4d98b8945243416a495bc4edc4299fa3c27d010dcada6e2"
Nov 01 07:15:02.803080 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:02.803053    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891955-jmdl8_odh-ai-gateway-infra(3ad1437a-ac0a-4501-a3ee-1cb870cd81f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9"
Nov 01 07:15:02.948773 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:02.948737    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:15:03.809141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:03.809109    4468 scope.go:122] "RemoveContainer" containerID="59d239291b17afe9c4d98b8945243416a495bc4edc4299fa3c27d010dcada6e2"
Nov 01 07:15:03.809570 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:03.809363    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891955-jmdl8_odh-ai-gateway-infra(3ad1437a-ac0a-4501-a3ee-1cb870cd81f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9"
Nov 01 07:15:14.589179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:14.589148    4468 scope.go:122] "RemoveContainer" containerID="59d239291b17afe9c4d98b8945243416a495bc4edc4299fa3c27d010dcada6e2"
Nov 01 07:15:15.878685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:15.878645    4468 generic.go:356] "Generic (PLEG): container finished" podID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerID="bb993231fb1c6673575b1d14e10dee1f91ee6ef605362988f8770a757c7fd874" exitCode=60
Nov 01 07:15:15.879214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:15.878722    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" event={"ID":"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9","Type":"ContainerDied","Data":"bb993231fb1c6673575b1d14e10dee1f91ee6ef605362988f8770a757c7fd874"}
Nov 01 07:15:15.879214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:15.878830    4468 scope.go:122] "RemoveContainer" containerID="59d239291b17afe9c4d98b8945243416a495bc4edc4299fa3c27d010dcada6e2"
Nov 01 07:15:15.879319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:15.879263    4468 scope.go:122] "RemoveContainer" containerID="bb993231fb1c6673575b1d14e10dee1f91ee6ef605362988f8770a757c7fd874"
Nov 01 07:15:15.879598 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:15.879573    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891955-jmdl8_odh-ai-gateway-infra(3ad1437a-ac0a-4501-a3ee-1cb870cd81f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9"
Nov 01 07:15:17.000975 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.000941    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"]
Nov 01 07:15:17.139844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.139816    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:17.310416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.310240    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm\" (UniqueName: \"kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm\") pod \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") "
Nov 01 07:15:17.310416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.310334    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca\") pod \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\" (UID: \"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9\") "
Nov 01 07:15:17.311210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.311179    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca" pod "3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" (UID: "3ad1437a-ac0a-4501-a3ee-1cb870cd81f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:17.313593 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.313549    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm" pod "3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" (UID: "3ad1437a-ac0a-4501-a3ee-1cb870cd81f9"). InnerVolumeSpecName "kube-api-access-zwqsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:17.411689 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.411648    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqsm\" (UniqueName: \"kubernetes.io/projected/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-kube-api-access-zwqsm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:17.411880 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.411680    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:17.891930 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.891887    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8" event={"ID":"3ad1437a-ac0a-4501-a3ee-1cb870cd81f9","Type":"ContainerDied","Data":"e8e7d0e097308810da1221bbd3cc06d43a3f26f2e1f3adc44ff74dcf90b09a54"}
Nov 01 07:15:17.891930 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.891920    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"
Nov 01 07:15:17.892212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:17.891945    4468 scope.go:122] "RemoveContainer" containerID="bb993231fb1c6673575b1d14e10dee1f91ee6ef605362988f8770a757c7fd874"
Nov 01 07:15:18.054176 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:18.054140    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"]
Nov 01 07:15:18.059809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:18.059778    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891955-jmdl8"]
Nov 01 07:15:18.589284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:18.589247    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" path="/var/lib/kubelet/pods/3ad1437a-ac0a-4501-a3ee-1cb870cd81f9/volumes"
Nov 01 07:15:19.431709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.431665    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:15:19.918217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.918172    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"]
Nov 01 07:15:19.918444 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.918416    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy" containerID="cri-o://103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da" gracePeriod=30
Nov 01 07:15:19.968573 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.968533    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"]
Nov 01 07:15:19.968785 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.968764    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy" containerID="cri-o://5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1" gracePeriod=30
Nov 01 07:15:19.970529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.970492    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"]
Nov 01 07:15:19.970747 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:19.970709    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy" containerID="cri-o://074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299" gracePeriod=30
Nov 01 07:15:20.017525 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:20.017486    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"]
Nov 01 07:15:20.017722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:20.017692    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy" containerID="cri-o://d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98" gracePeriod=30
Nov 01 07:15:25.253417 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.253379    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:15:25.289083 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289050    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289100    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289132    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289183    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289245    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289324    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289372    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289422    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289513    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs\") pod \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\" (UID: \"0b223fe5-8990-4a0e-bf45-dcff7130ee2c\") "
Nov 01 07:15:25.289893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289797    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.290006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.289956    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.290997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.290937    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.291186 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.291011    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:25.291291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.291188    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.291494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.291451    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.292187 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.292131    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "kube-api-access-z2qxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.292415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.292374    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.293543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.293455    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:15:25.296021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.295997    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy" pod "0b223fe5-8990-4a0e-bf45-dcff7130ee2c" (UID: "0b223fe5-8990-4a0e-bf45-dcff7130ee2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.303289 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.303250    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:15:25.332287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.332258    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:15:25.342313 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.342283    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:15:25.390880 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.390841    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.390890    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.391069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.390921    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391133    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.391270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391223    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.391270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391234    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.391270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391255    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.391441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391284    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391328    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391363    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.391441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391398    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391442    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391503    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391533    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391560    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391585    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391625    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391657    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391658    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391701    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.391750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391733    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert\") pod \"dfff3421-28a2-4b25-9282-002bcd960b4f\" (UID: \"dfff3421-28a2-4b25-9282-002bcd960b4f\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391773    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391807    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391837    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391863    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391905    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.391944    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392011    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo\") pod \"86c69c45-e08c-46c5-a024-c1bd27b60f95\" (UID: \"86c69c45-e08c-46c5-a024-c1bd27b60f95\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392037    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket\") pod \"35610e79-3429-44ff-bda1-db811e84b394\" (UID: \"35610e79-3429-44ff-bda1-db811e84b394\") "
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392203    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392406    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392603    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392625    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392641    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392644    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392658    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35610e79-3429-44ff-bda1-db811e84b394-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392673    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392687    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392701    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392714    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392728    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392742    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392756    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2qxz\" (UniqueName: \"kubernetes.io/projected/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-kube-api-access-z2qxz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392771    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b223fe5-8990-4a0e-bf45-dcff7130ee2c-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.392777    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393036    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393053    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393095    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393117    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393293    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393374    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393526    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393643    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.393871    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:15:25.394067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.394044    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:15:25.396283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.394414    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "kube-api-access-8xlgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.396283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.395927    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.396283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.396003    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.396940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.396911    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:15:25.397079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.396989    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.397079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.397004    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.397207 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.397083    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.397333 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.397310    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:15:25.397756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.397738    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl" pod "86c69c45-e08c-46c5-a024-c1bd27b60f95" (UID: "86c69c45-e08c-46c5-a024-c1bd27b60f95"). InnerVolumeSpecName "kube-api-access-cn9dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.398084 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.398061    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy" pod "dfff3421-28a2-4b25-9282-002bcd960b4f" (UID: "dfff3421-28a2-4b25-9282-002bcd960b4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:15:25.398364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.398341    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992" pod "35610e79-3429-44ff-bda1-db811e84b394" (UID: "35610e79-3429-44ff-bda1-db811e84b394"). InnerVolumeSpecName "kube-api-access-g7992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493796    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xlgn\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-kube-api-access-8xlgn\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493826    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35610e79-3429-44ff-bda1-db811e84b394-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493837    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493848    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493857    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493867    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493875    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.493879 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493884    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493893    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7992\" (UniqueName: \"kubernetes.io/projected/35610e79-3429-44ff-bda1-db811e84b394-kube-api-access-g7992\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493902    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493909    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493917    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493925    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493933    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn9dl\" (UniqueName: \"kubernetes.io/projected/86c69c45-e08c-46c5-a024-c1bd27b60f95-kube-api-access-cn9dl\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493942    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfff3421-28a2-4b25-9282-002bcd960b4f-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493949    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493957    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493965    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c69c45-e08c-46c5-a024-c1bd27b60f95-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493973    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c69c45-e08c-46c5-a024-c1bd27b60f95-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493981    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493990    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35610e79-3429-44ff-bda1-db811e84b394-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.493999    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c69c45-e08c-46c5-a024-c1bd27b60f95-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.494272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.494006    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfff3421-28a2-4b25-9282-002bcd960b4f-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:15:25.936984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.936945    4468 generic.go:356] "Generic (PLEG): container finished" podID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerID="d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98" exitCode=0
Nov 01 07:15:25.937198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.937017    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"
Nov 01 07:15:25.937198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.937038    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" event={"ID":"dfff3421-28a2-4b25-9282-002bcd960b4f","Type":"ContainerDied","Data":"d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98"}
Nov 01 07:15:25.937198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.937142    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr" event={"ID":"dfff3421-28a2-4b25-9282-002bcd960b4f","Type":"ContainerDied","Data":"16893c2dcc4f31a5f0f2ca8ddc2f8d8f83cd5580bcf64d8861af80e095cbf9b8"}
Nov 01 07:15:25.937198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.937176    4468 scope.go:122] "RemoveContainer" containerID="d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98"
Nov 01 07:15:25.940059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.940032    4468 generic.go:356] "Generic (PLEG): container finished" podID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerID="103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da" exitCode=0
Nov 01 07:15:25.940278 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.940077    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" event={"ID":"0b223fe5-8990-4a0e-bf45-dcff7130ee2c","Type":"ContainerDied","Data":"103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da"}
Nov 01 07:15:25.940410 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.940324    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x" event={"ID":"0b223fe5-8990-4a0e-bf45-dcff7130ee2c","Type":"ContainerDied","Data":"735022ce51364fa7324f1dacec8fd6e4b32a2954b07d6af6119dac1ec19db9ed"}
Nov 01 07:15:25.940410 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.940140    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"
Nov 01 07:15:25.942031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.942007    4468 generic.go:356] "Generic (PLEG): container finished" podID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerID="074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299" exitCode=0
Nov 01 07:15:25.942172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.942128    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" event={"ID":"86c69c45-e08c-46c5-a024-c1bd27b60f95","Type":"ContainerDied","Data":"074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299"}
Nov 01 07:15:25.942237 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.942176    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9" event={"ID":"86c69c45-e08c-46c5-a024-c1bd27b60f95","Type":"ContainerDied","Data":"7eccd5d95f8989abe5d1a285aa40ac7ba58d015222dbb25025b7e514c5a52431"}
Nov 01 07:15:25.942296 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.942260    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"
Nov 01 07:15:25.946797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.946405    4468 generic.go:356] "Generic (PLEG): container finished" podID="35610e79-3429-44ff-bda1-db811e84b394" containerID="5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1" exitCode=0
Nov 01 07:15:25.946797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.946451    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" event={"ID":"35610e79-3429-44ff-bda1-db811e84b394","Type":"ContainerDied","Data":"5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1"}
Nov 01 07:15:25.946797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.946455    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"
Nov 01 07:15:25.946797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.946520    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx" event={"ID":"35610e79-3429-44ff-bda1-db811e84b394","Type":"ContainerDied","Data":"da8831744d20c106607d0eea50afb4ca51d27d19eb1b53a67e3bbbac7202fcf6"}
Nov 01 07:15:25.972161 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.972139    4468 scope.go:122] "RemoveContainer" containerID="d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98"
Nov 01 07:15:25.972600 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:25.972579    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98\": container with ID starting with d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98 not found: ID does not exist" containerID="d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98"
Nov 01 07:15:25.972679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.972611    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98"} err="failed to get container status \"d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98\": rpc error: code = NotFound desc = could not find container \"d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98\": container with ID starting with d826738347a71cfefbea034c12faf09764260c49eb4ee856d9d8d01b16c78f98 not found: ID does not exist"
Nov 01 07:15:25.972679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.972631    4468 scope.go:122] "RemoveContainer" containerID="103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da"
Nov 01 07:15:25.981679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.981661    4468 scope.go:122] "RemoveContainer" containerID="103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da"
Nov 01 07:15:25.981999 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:25.981975    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da\": container with ID starting with 103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da not found: ID does not exist" containerID="103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da"
Nov 01 07:15:25.982071 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.982013    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da"} err="failed to get container status \"103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da\": rpc error: code = NotFound desc = could not find container \"103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da\": container with ID starting with 103f7d1dd1b315f4a45611979ed6ad7a7ba7a7e79b0178d4ce2dfc474aeca5da not found: ID does not exist"
Nov 01 07:15:25.982071 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.982041    4468 scope.go:122] "RemoveContainer" containerID="074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299"
Nov 01 07:15:25.990915 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.990893    4468 scope.go:122] "RemoveContainer" containerID="074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299"
Nov 01 07:15:25.991203 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:25.991187    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299\": container with ID starting with 074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299 not found: ID does not exist" containerID="074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299"
Nov 01 07:15:25.991248 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.991213    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299"} err="failed to get container status \"074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299\": rpc error: code = NotFound desc = could not find container \"074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299\": container with ID starting with 074971b57abf4663711a0871a91fcfea84988567bfb85f14650f9da9a56a3299 not found: ID does not exist"
Nov 01 07:15:25.991248 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:25.991230    4468 scope.go:122] "RemoveContainer" containerID="5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1"
Nov 01 07:15:26.002922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.001640    4468 scope.go:122] "RemoveContainer" containerID="5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1"
Nov 01 07:15:26.003439 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:26.003418    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1\": container with ID starting with 5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1 not found: ID does not exist" containerID="5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1"
Nov 01 07:15:26.003549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.003449    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1"} err="failed to get container status \"5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1\": rpc error: code = NotFound desc = could not find container \"5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1\": container with ID starting with 5ab662d1ed6cb2b5b0de94b5ddb35c9c0df671167021141e8aa60e8f4d2566e1 not found: ID does not exist"
Nov 01 07:15:26.100710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.100669    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"]
Nov 01 07:15:26.103836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.103800    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-j4f5x"]
Nov 01 07:15:26.212261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.212120    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"]
Nov 01 07:15:26.215034 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.215000    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-sjcrx"]
Nov 01 07:15:26.323788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.323752    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"]
Nov 01 07:15:26.328580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.328549    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-zq7gr"]
Nov 01 07:15:26.437539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.437503    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"]
Nov 01 07:15:26.441087 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.441048    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xqsk9"]
Nov 01 07:15:26.589127 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.589091    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" path="/var/lib/kubelet/pods/0b223fe5-8990-4a0e-bf45-dcff7130ee2c/volumes"
Nov 01 07:15:26.589544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.589530    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35610e79-3429-44ff-bda1-db811e84b394" path="/var/lib/kubelet/pods/35610e79-3429-44ff-bda1-db811e84b394/volumes"
Nov 01 07:15:26.589919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.589907    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" path="/var/lib/kubelet/pods/86c69c45-e08c-46c5-a024-c1bd27b60f95/volumes"
Nov 01 07:15:26.590626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:26.590610    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" path="/var/lib/kubelet/pods/dfff3421-28a2-4b25-9282-002bcd960b4f/volumes"
Nov 01 07:15:32.953041 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:15:32.952994    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:15:40.659996 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.659957    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"]
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660333    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660344    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660358    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660363    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660372    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660377    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660387    4468 cpu_manager.go:414] "containerMap: removing container" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy"
Nov 01 07:15:40.660400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660392    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660414    4468 cpu_manager.go:414] "containerMap: removing container" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660419    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660429    4468 cpu_manager.go:414] "containerMap: removing container" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660433    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e262958f-fc5e-4b5d-b231-775478f412b8" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660441    4468 cpu_manager.go:414] "containerMap: removing container" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660445    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660527    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b223fe5-8990-4a0e-bf45-dcff7130ee2c" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660535    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660541    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660549    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfff3421-28a2-4b25-9282-002bcd960b4f" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660559    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c69c45-e08c-46c5-a024-c1bd27b60f95" containerName="istio-proxy"
Nov 01 07:15:40.660704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.660564    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="35610e79-3429-44ff-bda1-db811e84b394" containerName="istio-proxy"
Nov 01 07:15:40.665422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.665400    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.721025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.720994    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"]
Nov 01 07:15:40.743251 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.743211    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.743441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.743273    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6sw\" (UniqueName: \"kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.743441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.743340    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.743441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.743411    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.743636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.743455    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.743636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.743569    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.767873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.767838    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"kube-root-ca.crt\""
Nov 01 07:15:40.767873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.767838    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"openshift-service-ca.crt\""
Nov 01 07:15:40.767873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.767870    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"premium-main-976689-kserve-self-signed-certs\""
Nov 01 07:15:40.768076 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.767854    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"default-dockercfg-zbj4g\""
Nov 01 07:15:40.844196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844161    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6sw\" (UniqueName: \"kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844194    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844212    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844243    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844291    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844325    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844684    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844778 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844760    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.844778 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.844771    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.846822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.846795    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.847021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.847006    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.900833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.900796    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6sw\" (UniqueName: \"kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw\") pod \"premium-main-976689-kserve-69ddf894f-q8ddr\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") " pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:40.976808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:40.976721    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:41.348978 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:41.348948    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"]
Nov 01 07:15:41.350949 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:15:41.350918    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1307d6e_a515_4e2b_b1cb_7b462cbb1c2f.slice/crio-a1ba448428fadf76c1c88515578f11f845a720c3f58d055c7a5e28dd827b32a2 WatchSource:0}: Error finding container a1ba448428fadf76c1c88515578f11f845a720c3f58d055c7a5e28dd827b32a2: Status 404 returned error can't find the container with id a1ba448428fadf76c1c88515578f11f845a720c3f58d055c7a5e28dd827b32a2
Nov 01 07:15:42.033412 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:42.033370    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr" event={"ID":"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f","Type":"ContainerStarted","Data":"a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef"}
Nov 01 07:15:42.033412 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:42.033420    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr" event={"ID":"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f","Type":"ContainerStarted","Data":"a1ba448428fadf76c1c88515578f11f845a720c3f58d055c7a5e28dd827b32a2"}
Nov 01 07:15:42.178761 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:42.178705    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr" podStartSLOduration=2.178690223 podStartE2EDuration="2.178690223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:15:42.176215334 +0000 UTC m=+3953.744022187" watchObservedRunningTime="2026-11-01 07:15:42.178690223 +0000 UTC m=+3953.746497065"
Nov 01 07:15:42.977428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:42.977387    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:45.567610 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:45.567569    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:15:50.977177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:50.977132    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:50.989843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:50.989816    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:51.090154 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:51.090126    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:15:52.573461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:52.573428    4468 scope.go:122] "RemoveContainer" containerID="a9e6f7bd083e4b4eaf81232a76ea9191096dbb5b2dd756d56032d08f85f07d3f"
Nov 01 07:15:52.582174 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:52.582155    4468 scope.go:122] "RemoveContainer" containerID="5f4b1c3159ede3797ff735374d3d6a55ec2b9eb02f8ba383070b8bee1ee002d1"
Nov 01 07:15:52.590477 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:15:52.590444    4468 scope.go:122] "RemoveContainer" containerID="4a6386f71ad90d4ab72d2e3266daeda098651f3cdda57757dd5cc6120cb8f8bb"
Nov 01 07:16:02.947394 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:16:02.947357    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:16:11.717550 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:11.717515    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:16:30.818260 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:30.818163    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-z7whx"]
Nov 01 07:16:30.818752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:30.818457    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64d475bcc9-z7whx" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager" containerID="cri-o://c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4" gracePeriod=30
Nov 01 07:16:31.164166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.164141    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 07:16:31.229518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.229455    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert\") pod \"3294a44f-c41c-46fa-9844-842944e0ad5e\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") "
Nov 01 07:16:31.229725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.229533    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr\" (UniqueName: \"kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr\") pod \"3294a44f-c41c-46fa-9844-842944e0ad5e\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") "
Nov 01 07:16:31.229725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.229562    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs\") pod \"3294a44f-c41c-46fa-9844-842944e0ad5e\" (UID: \"3294a44f-c41c-46fa-9844-842944e0ad5e\") "
Nov 01 07:16:31.231843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.231810    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr" pod "3294a44f-c41c-46fa-9844-842944e0ad5e" (UID: "3294a44f-c41c-46fa-9844-842944e0ad5e"). InnerVolumeSpecName "kube-api-access-xh7tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:16:31.231843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.231819    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert" pod "3294a44f-c41c-46fa-9844-842944e0ad5e" (UID: "3294a44f-c41c-46fa-9844-842944e0ad5e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:16:31.231843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.231839    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs" pod "3294a44f-c41c-46fa-9844-842944e0ad5e" (UID: "3294a44f-c41c-46fa-9844-842944e0ad5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:16:31.270722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.270700    4468 generic.go:356] "Generic (PLEG): container finished" podID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerID="c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4" exitCode=0
Nov 01 07:16:31.270832 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.270753    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-z7whx"
Nov 01 07:16:31.270832 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.270775    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-z7whx" event={"ID":"3294a44f-c41c-46fa-9844-842944e0ad5e","Type":"ContainerDied","Data":"c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4"}
Nov 01 07:16:31.270913 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.270833    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-z7whx" event={"ID":"3294a44f-c41c-46fa-9844-842944e0ad5e","Type":"ContainerDied","Data":"c175d7d47798564a827f88880174097755f6a0d97e50902602f278e4be67a393"}
Nov 01 07:16:31.270913 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.270856    4468 scope.go:122] "RemoveContainer" containerID="c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4"
Nov 01 07:16:31.282221 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.282200    4468 scope.go:122] "RemoveContainer" containerID="f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"
Nov 01 07:16:31.291559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.291548    4468 scope.go:122] "RemoveContainer" containerID="c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4"
Nov 01 07:16:31.291838 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:16:31.291815    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4\": container with ID starting with c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4 not found: ID does not exist" containerID="c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4"
Nov 01 07:16:31.291926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.291848    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4"} err="failed to get container status \"c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4\": rpc error: code = NotFound desc = could not find container \"c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4\": container with ID starting with c3099f28c29dd4eb15153171bcbb8857493928a95d10c586f50eabab791842a4 not found: ID does not exist"
Nov 01 07:16:31.291926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.291876    4468 scope.go:122] "RemoveContainer" containerID="f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"
Nov 01 07:16:31.292143 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:16:31.292119    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad\": container with ID starting with f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad not found: ID does not exist" containerID="f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"
Nov 01 07:16:31.292220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.292153    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad"} err="failed to get container status \"f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad\": rpc error: code = NotFound desc = could not find container \"f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad\": container with ID starting with f6cebda75bfcd6030155cab9f16fe037f60932475c92fadb8cf12621cf12fbad not found: ID does not exist"
Nov 01 07:16:31.330621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.330597    4468 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-webhook-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:31.330621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.330621    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7tr\" (UniqueName: \"kubernetes.io/projected/3294a44f-c41c-46fa-9844-842944e0ad5e-kube-api-access-xh7tr\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:31.330791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.330630    4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3294a44f-c41c-46fa-9844-842944e0ad5e-metrics-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:31.440499 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.440400    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-z7whx"]
Nov 01 07:16:31.445475 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:31.445440    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-z7whx"]
Nov 01 07:16:32.284061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284029    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d475bcc9-m8vd5"]
Nov 01 07:16:32.284452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284389    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager"
Nov 01 07:16:32.284452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284400    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager"
Nov 01 07:16:32.284452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284434    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:16:32.284452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284439    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:16:32.284691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284519    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager"
Nov 01 07:16:32.284691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.284529    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad1437a-ac0a-4501-a3ee-1cb870cd81f9" containerName="cleanup"
Nov 01 07:16:32.288753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.288735    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.341226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.341194    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 07:16:32.390970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.390934    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8r64\""
Nov 01 07:16:32.390970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.390951    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 07:16:32.390970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.390956    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 07:16:32.392951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.392929    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-m8vd5"]
Nov 01 07:16:32.440808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.440778    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwrs\" (UniqueName: \"kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.440950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.440828    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.441016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.440948    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.541575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.541460    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwrs\" (UniqueName: \"kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.541718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.541572    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.541718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.541645    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.544170 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.544151    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.544286 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.544265    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.589125 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.589092    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" path="/var/lib/kubelet/pods/3294a44f-c41c-46fa-9844-842944e0ad5e/volumes"
Nov 01 07:16:32.598080 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.598046    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwrs\" (UniqueName: \"kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs\") pod \"maas-controller-64d475bcc9-m8vd5\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") " pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.600908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.600890    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:32.946561 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:16:32.946518    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:16:32.970981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:32.970884    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-m8vd5"]
Nov 01 07:16:33.005841 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:16:33.005816    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15667a56_6b92_4787_ac98_f633cadbe7df.slice/crio-e5ec7fdac344916cc6bbaec4640c279d40e7e4844f4645c64217c43423482ac4 WatchSource:0}: Error finding container e5ec7fdac344916cc6bbaec4640c279d40e7e4844f4645c64217c43423482ac4: Status 404 returned error can't find the container with id e5ec7fdac344916cc6bbaec4640c279d40e7e4844f4645c64217c43423482ac4
Nov 01 07:16:33.007053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:33.007036    4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 07:16:33.284533 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:33.284442    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-m8vd5" event={"ID":"15667a56-6b92-4787-ac98-f633cadbe7df","Type":"ContainerStarted","Data":"e5ec7fdac344916cc6bbaec4640c279d40e7e4844f4645c64217c43423482ac4"}
Nov 01 07:16:34.291918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:34.291829    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-m8vd5" event={"ID":"15667a56-6b92-4787-ac98-f633cadbe7df","Type":"ContainerStarted","Data":"1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b"}
Nov 01 07:16:34.292338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:34.291936    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:34.404979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:34.404917    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d475bcc9-m8vd5" podStartSLOduration=1.506635129 podStartE2EDuration="2.404897916s" totalImagesPullingTime="898.262787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:16:34.404564092 +0000 UTC m=+4005.972370923" watchObservedRunningTime="2026-11-01 07:16:34.404897916 +0000 UTC m=+4005.972704747"
Nov 01 07:16:45.299433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:45.299401    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:16:50.613179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.613148    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"]
Nov 01 07:16:50.613632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.613588    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager"
Nov 01 07:16:50.613632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.613600    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager"
Nov 01 07:16:50.613712 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.613669    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3294a44f-c41c-46fa-9844-842944e0ad5e" containerName="manager"
Nov 01 07:16:50.616870 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.616855    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.662858 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.662825    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"]
Nov 01 07:16:50.666320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.666305    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.677179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.677159    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 07:16:50.677297 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.677222    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g6wqc\""
Nov 01 07:16:50.713916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.713879    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.713916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.713916    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txgd\" (UniqueName: \"kubernetes.io/projected/8dfde8ce-8829-4310-a77e-0e64d77c0c45-kube-api-access-4txgd\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714156 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.713934    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714156 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.713955    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714156 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.714120    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714323 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.714159    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714323 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.714275    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.714332    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.714440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.714369    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.718772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.718744    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"]
Nov 01 07:16:50.769848 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.769804    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"]
Nov 01 07:16:50.815397 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815363    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.815397 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815414    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.815698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815538    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.815698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815576    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.815698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815656    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.815698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815691    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815836    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815872    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815880    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815894    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815926    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815944    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7adc820-4dbf-455b-ada4-9b655c77c786-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.815983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.815973    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816088    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816133    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816164    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txgd\" (UniqueName: \"kubernetes.io/projected/8dfde8ce-8829-4310-a77e-0e64d77c0c45-kube-api-access-4txgd\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816175    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816213    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816227    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816286    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm55k\" (UniqueName: \"kubernetes.io/projected/a7adc820-4dbf-455b-ada4-9b655c77c786-kube-api-access-sm55k\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816324    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816410    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.816844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.816826    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.818589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.818572    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.818651 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.818630    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.871936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.871863    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfde8ce-8829-4310-a77e-0e64d77c0c45-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.872085 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.871947    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txgd\" (UniqueName: \"kubernetes.io/projected/8dfde8ce-8829-4310-a77e-0e64d77c0c45-kube-api-access-4txgd\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hjj86\" (UID: \"8dfde8ce-8829-4310-a77e-0e64d77c0c45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.917762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.917728    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.917917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.917771    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.917997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.917976    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918087 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918069    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918094    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918122    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7adc820-4dbf-455b-ada4-9b655c77c786-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918155    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918221    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918254    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918255    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm55k\" (UniqueName: \"kubernetes.io/projected/a7adc820-4dbf-455b-ada4-9b655c77c786-kube-api-access-sm55k\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918322    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918538    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918595    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.918782 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.918765    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7adc820-4dbf-455b-ada4-9b655c77c786-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.920538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.920509    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.920672 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.920655    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.929111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.929093    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:50.973998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.973969    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm55k\" (UniqueName: \"kubernetes.io/projected/a7adc820-4dbf-455b-ada4-9b655c77c786-kube-api-access-sm55k\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.973998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.973989    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7adc820-4dbf-455b-ada4-9b655c77c786-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qb2mm\" (UID: \"a7adc820-4dbf-455b-ada4-9b655c77c786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:50.979191 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:50.978199    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:51.349833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.349803    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"]
Nov 01 07:16:51.351323 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:16:51.351298    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfde8ce_8829_4310_a77e_0e64d77c0c45.slice/crio-c49c25c1c075ab3e88b17e5320d92329838ac7c159263132cc5433118290f9f6 WatchSource:0}: Error finding container c49c25c1c075ab3e88b17e5320d92329838ac7c159263132cc5433118290f9f6: Status 404 returned error can't find the container with id c49c25c1c075ab3e88b17e5320d92329838ac7c159263132cc5433118290f9f6
Nov 01 07:16:51.353692 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.353659    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:16:51.353788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.353726    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:16:51.353788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.353754    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:16:51.375093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.375065    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" event={"ID":"8dfde8ce-8829-4310-a77e-0e64d77c0c45","Type":"ContainerStarted","Data":"c49c25c1c075ab3e88b17e5320d92329838ac7c159263132cc5433118290f9f6"}
Nov 01 07:16:51.403866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.403793    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"]
Nov 01 07:16:51.405933 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:16:51.405909    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7adc820_4dbf_455b_ada4_9b655c77c786.slice/crio-84ab25156770ef33c7df10bfce2fcf01810657a68be7639f94ec7a122db3f578 WatchSource:0}: Error finding container 84ab25156770ef33c7df10bfce2fcf01810657a68be7639f94ec7a122db3f578: Status 404 returned error can't find the container with id 84ab25156770ef33c7df10bfce2fcf01810657a68be7639f94ec7a122db3f578
Nov 01 07:16:51.408503 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.408448    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:16:51.408575 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.408560    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:16:51.408618 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:51.408596    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:16:52.381659 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.381627    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" event={"ID":"a7adc820-4dbf-455b-ada4-9b655c77c786","Type":"ContainerStarted","Data":"48b83e2c9a79b4d6ac8129d49e4d1e1c5621fe41f66a996dd02f7e8af2c40a01"}
Nov 01 07:16:52.382196 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.381677    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" event={"ID":"a7adc820-4dbf-455b-ada4-9b655c77c786","Type":"ContainerStarted","Data":"84ab25156770ef33c7df10bfce2fcf01810657a68be7639f94ec7a122db3f578"}
Nov 01 07:16:52.383226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.383204    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" event={"ID":"8dfde8ce-8829-4310-a77e-0e64d77c0c45","Type":"ContainerStarted","Data":"ae87003c1e5ff827601882006665f3ffdd1fd149e90f227a49d054cfb68a6300"}
Nov 01 07:16:52.602227 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.602178    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podStartSLOduration=2.602164969 podStartE2EDuration="2.602164969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:16:52.552372969 +0000 UTC m=+4024.120179825" watchObservedRunningTime="2026-11-01 07:16:52.602164969 +0000 UTC m=+4024.169971798"
Nov 01 07:16:52.679505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.679434    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podStartSLOduration=2.679411924 podStartE2EDuration="2.679411924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:16:52.676669416 +0000 UTC m=+4024.244476246" watchObservedRunningTime="2026-11-01 07:16:52.679411924 +0000 UTC m=+4024.247218753"
Nov 01 07:16:52.932552 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.932397    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:16:52.932752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.932640    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:52.932752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.932690    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:52.979130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.979098    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:16:52.980038 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.980020    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:52.980092 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:52.980056    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:53.929921 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:53.929878    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:53.930619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:53.930576    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:53.979566 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:53.979532    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:53.979810 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:53.979791    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:54.933906 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:54.933866    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:54.934577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:54.934545    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:54.979700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:54.979666    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:54.979934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:54.979910    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:55.169786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.169748    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"]
Nov 01 07:16:55.173298 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.173270    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.181524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.181499    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.181761 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.181747    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttgz\" (UniqueName: \"kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.181956 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.181938    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.243626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.243540    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"]
Nov 01 07:16:55.244389 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.244367    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:16:55.244644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.244628    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:16:55.284492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.283376    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.284492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.283414    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttgz\" (UniqueName: \"kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.284492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.283455    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.284492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.284344    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:16:55.284492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.284417    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:16:55.286041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.286021    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.294808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.294785    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.365236 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.365202    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttgz\" (UniqueName: \"kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz\") pod \"maas-api-revoke-sub-85237e45-xxlsp\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.490911 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.490876    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:55.919502 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.915021    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"]
Nov 01 07:16:55.932422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.931977    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:55.932422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.932037    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:55.979247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.979222    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:55.979609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:55.979273    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:56.410527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.410459    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp" event={"ID":"25ea60f1-02b2-4556-8b1f-883ecd3957c7","Type":"ContainerStarted","Data":"80b7b9736a26f3b7a95d13d8504a1bd603e047c83278b5b37bec1290dbab7969"}
Nov 01 07:16:56.410840 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.410815    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp" event={"ID":"25ea60f1-02b2-4556-8b1f-883ecd3957c7","Type":"ContainerStarted","Data":"660fad2100f1f071b09d0def724c6cfbc014116d224c17273e0cc7507c5e1e81"}
Nov 01 07:16:56.532834 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.532704    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp" podStartSLOduration=1.532683351 podStartE2EDuration="1.532683351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:16:56.529000331 +0000 UTC m=+4028.096807162" watchObservedRunningTime="2026-11-01 07:16:56.532683351 +0000 UTC m=+4028.100490182"
Nov 01 07:16:56.929887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.929841    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:56.930191 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.930164    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:56.979679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.979580    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:56.979679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:56.979629    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:57.444509 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:57.444400    4468 generic.go:356] "Generic (PLEG): container finished" podID="25ea60f1-02b2-4556-8b1f-883ecd3957c7" containerID="80b7b9736a26f3b7a95d13d8504a1bd603e047c83278b5b37bec1290dbab7969" exitCode=0
Nov 01 07:16:57.444737 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:57.444575    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp" event={"ID":"25ea60f1-02b2-4556-8b1f-883ecd3957c7","Type":"ContainerDied","Data":"80b7b9736a26f3b7a95d13d8504a1bd603e047c83278b5b37bec1290dbab7969"}
Nov 01 07:16:57.929653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:57.929611    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:57.929826 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:57.929697    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:57.979988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:57.979948    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:57.980387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:57.980014    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:58.588154 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.588124    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:58.620239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.620205    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca\") pod \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") "
Nov 01 07:16:58.620513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.620274    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz\" (UniqueName: \"kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz\") pod \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") "
Nov 01 07:16:58.620513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.620393    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca\") pod \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\" (UID: \"25ea60f1-02b2-4556-8b1f-883ecd3957c7\") "
Nov 01 07:16:58.620875 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.620842    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca" pod "25ea60f1-02b2-4556-8b1f-883ecd3957c7" (UID: "25ea60f1-02b2-4556-8b1f-883ecd3957c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:16:58.622669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.622640    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz" pod "25ea60f1-02b2-4556-8b1f-883ecd3957c7" (UID: "25ea60f1-02b2-4556-8b1f-883ecd3957c7"). InnerVolumeSpecName "kube-api-access-mttgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:16:58.622794 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.622721    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca" pod "25ea60f1-02b2-4556-8b1f-883ecd3957c7" (UID: "25ea60f1-02b2-4556-8b1f-883ecd3957c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:16:58.722051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.721954    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:58.722051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.721990    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mttgz\" (UniqueName: \"kubernetes.io/projected/25ea60f1-02b2-4556-8b1f-883ecd3957c7-kube-api-access-mttgz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:58.722051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.722001    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ea60f1-02b2-4556-8b1f-883ecd3957c7-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:16:58.929802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.929757    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 01 07:16:58.929995 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.929823    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 01 07:16:58.979872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.979772    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 01 07:16:58.979872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:58.979841    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 01 07:16:59.458895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:59.458859    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"
Nov 01 07:16:59.458895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:59.458884    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp" event={"ID":"25ea60f1-02b2-4556-8b1f-883ecd3957c7","Type":"ContainerDied","Data":"660fad2100f1f071b09d0def724c6cfbc014116d224c17273e0cc7507c5e1e81"}
Nov 01 07:16:59.459191 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:16:59.458943    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660fad2100f1f071b09d0def724c6cfbc014116d224c17273e0cc7507c5e1e81"
Nov 01 07:17:00.970690 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:00.970645    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:17:00.971215 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:00.970722    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:17:01.020668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:01.020626    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:17:01.020950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:01.020918    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:17:02.012665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.012630    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:17:02.013214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.012729    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:17:02.061622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.061584    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:17:02.061847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.061639    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:17:02.286350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.286253    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:17:02.286860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.286800    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:17:02.287880 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.287851    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm"
Nov 01 07:17:02.438147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.438120    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:17:02.438446 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.438427    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:17:02.439062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:02.439046    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86"
Nov 01 07:17:02.946774 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:17:02.946741    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:17:05.738434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.738392    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"]
Nov 01 07:17:05.739011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.738821    4468 cpu_manager.go:414] "containerMap: removing container" podUID="25ea60f1-02b2-4556-8b1f-883ecd3957c7" containerName="revoke-keys"
Nov 01 07:17:05.739011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.738832    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ea60f1-02b2-4556-8b1f-883ecd3957c7" containerName="revoke-keys"
Nov 01 07:17:05.739011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.738953    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ea60f1-02b2-4556-8b1f-883ecd3957c7" containerName="revoke-keys"
Nov 01 07:17:05.742505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.742488    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800256 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800222    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800268    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800335    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dlvr\" (UniqueName: \"kubernetes.io/projected/c3026d10-479c-4eaa-a18f-d2a83a7b508d-kube-api-access-8dlvr\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800392    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800411    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800514    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800534    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800564    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.800809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.800611    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.841123 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.841092    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"]
Nov 01 07:17:05.901942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.901905    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.901942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.901947    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.901969    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dlvr\" (UniqueName: \"kubernetes.io/projected/c3026d10-479c-4eaa-a18f-d2a83a7b508d-kube-api-access-8dlvr\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.901999    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902018    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902038    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902059    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902093    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902119    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902383    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902416    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902747 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902690    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.902870 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.902834    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.903289 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.903264    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.904798 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.904774    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.904902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.904884    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.958915 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.958878    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3026d10-479c-4eaa-a18f-d2a83a7b508d-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:05.959082 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:05.958989    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dlvr\" (UniqueName: \"kubernetes.io/projected/c3026d10-479c-4eaa-a18f-d2a83a7b508d-kube-api-access-8dlvr\") pod \"maas-default-gateway-openshift-default-7d9897b54c-pn2tc\" (UID: \"c3026d10-479c-4eaa-a18f-d2a83a7b508d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:06.055417 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:06.055383    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:06.480198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:06.480124    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"]
Nov 01 07:17:06.483268 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:17:06.483239    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3026d10_479c_4eaa_a18f_d2a83a7b508d.slice/crio-57707e81a05cb81ec5e2207257cc1a3a4305421431be0b60376ec61a86b13aaa WatchSource:0}: Error finding container 57707e81a05cb81ec5e2207257cc1a3a4305421431be0b60376ec61a86b13aaa: Status 404 returned error can't find the container with id 57707e81a05cb81ec5e2207257cc1a3a4305421431be0b60376ec61a86b13aaa
Nov 01 07:17:06.485487 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:06.485430    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:17:06.485590 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:06.485547    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:17:06.485590 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:06.485578    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:17:06.495651 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:06.495620    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" event={"ID":"c3026d10-479c-4eaa-a18f-d2a83a7b508d","Type":"ContainerStarted","Data":"57707e81a05cb81ec5e2207257cc1a3a4305421431be0b60376ec61a86b13aaa"}
Nov 01 07:17:07.501650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:07.501606    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" event={"ID":"c3026d10-479c-4eaa-a18f-d2a83a7b508d","Type":"ContainerStarted","Data":"f065443fbf10a4ce4e7c318e9fc9975820f5e86efaadc390982f193785f2e6b7"}
Nov 01 07:17:07.627958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:07.627853    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-m8vd5"]
Nov 01 07:17:07.628177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:07.628114    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64d475bcc9-m8vd5" podUID="15667a56-6b92-4787-ac98-f633cadbe7df" containerName="manager" containerID="cri-o://1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b" gracePeriod=30
Nov 01 07:17:07.672100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:07.672046    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podStartSLOduration=2.672031782 podStartE2EDuration="2.672031782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:17:07.622385127 +0000 UTC m=+4039.190191959" watchObservedRunningTime="2026-11-01 07:17:07.672031782 +0000 UTC m=+4039.239838612"
Nov 01 07:17:07.892825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:07.892799    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:17:08.035536 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.035496    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs\" (UniqueName: \"kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs\") pod \"15667a56-6b92-4787-ac98-f633cadbe7df\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") "
Nov 01 07:17:08.035753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.035560    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert\") pod \"15667a56-6b92-4787-ac98-f633cadbe7df\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") "
Nov 01 07:17:08.035753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.035612    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs\") pod \"15667a56-6b92-4787-ac98-f633cadbe7df\" (UID: \"15667a56-6b92-4787-ac98-f633cadbe7df\") "
Nov 01 07:17:08.038092 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.038061    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs" pod "15667a56-6b92-4787-ac98-f633cadbe7df" (UID: "15667a56-6b92-4787-ac98-f633cadbe7df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:17:08.038210 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.038107    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs" pod "15667a56-6b92-4787-ac98-f633cadbe7df" (UID: "15667a56-6b92-4787-ac98-f633cadbe7df"). InnerVolumeSpecName "kube-api-access-wzwrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:17:08.038876 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.038837    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert" pod "15667a56-6b92-4787-ac98-f633cadbe7df" (UID: "15667a56-6b92-4787-ac98-f633cadbe7df"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:17:08.055857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.055788    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:08.057132 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.057022    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 01 07:17:08.057132 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.057076    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 01 07:17:08.136738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.136708    4468 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-webhook-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:08.136738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.136734    4468 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15667a56-6b92-4787-ac98-f633cadbe7df-metrics-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:08.136738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.136744    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwrs\" (UniqueName: \"kubernetes.io/projected/15667a56-6b92-4787-ac98-f633cadbe7df-kube-api-access-wzwrs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:17:08.530457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.530361    4468 generic.go:356] "Generic (PLEG): container finished" podID="15667a56-6b92-4787-ac98-f633cadbe7df" containerID="1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b" exitCode=0
Nov 01 07:17:08.530457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.530437    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-m8vd5"
Nov 01 07:17:08.530986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.530453    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-m8vd5" event={"ID":"15667a56-6b92-4787-ac98-f633cadbe7df","Type":"ContainerDied","Data":"1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b"}
Nov 01 07:17:08.530986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.530565    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-m8vd5" event={"ID":"15667a56-6b92-4787-ac98-f633cadbe7df","Type":"ContainerDied","Data":"e5ec7fdac344916cc6bbaec4640c279d40e7e4844f4645c64217c43423482ac4"}
Nov 01 07:17:08.530986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.530588    4468 scope.go:122] "RemoveContainer" containerID="1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b"
Nov 01 07:17:08.544061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.544034    4468 scope.go:122] "RemoveContainer" containerID="1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b"
Nov 01 07:17:08.544375 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:17:08.544357    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b\": container with ID starting with 1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b not found: ID does not exist" containerID="1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b"
Nov 01 07:17:08.544454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.544383    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b"} err="failed to get container status \"1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b\": rpc error: code = NotFound desc = could not find container \"1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b\": container with ID starting with 1af97eeda68bee6269a4aefe59004236e4e043cb2c6906be8aa4325ffc94819b not found: ID does not exist"
Nov 01 07:17:08.693397 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.693370    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-m8vd5"]
Nov 01 07:17:08.742398 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:08.742349    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-m8vd5"]
Nov 01 07:17:09.056800 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:09.056764    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 01 07:17:09.056993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:09.056819    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 01 07:17:10.056183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.056140    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 01 07:17:10.056713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.056197    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 01 07:17:10.079401 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.079369    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d475bcc9-cs8xr"]
Nov 01 07:17:10.079886 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.079867    4468 cpu_manager.go:414] "containerMap: removing container" podUID="15667a56-6b92-4787-ac98-f633cadbe7df" containerName="manager"
Nov 01 07:17:10.080003 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.079888    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15667a56-6b92-4787-ac98-f633cadbe7df" containerName="manager"
Nov 01 07:17:10.080062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.080031    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="15667a56-6b92-4787-ac98-f633cadbe7df" containerName="manager"
Nov 01 07:17:10.084539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.084518    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.137853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.137814    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 07:17:10.157431 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.157399    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba3540-a6bb-464f-89b1-0288581d1d47-metrics-certs\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.157570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.157457    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3ba3540-a6bb-464f-89b1-0288581d1d47-webhook-cert\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.157622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.157571    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zq7b\" (UniqueName: \"kubernetes.io/projected/b3ba3540-a6bb-464f-89b1-0288581d1d47-kube-api-access-9zq7b\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.187445 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.187422    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 07:17:10.187985 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.187965    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8r64\""
Nov 01 07:17:10.188433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.188416    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 07:17:10.190994 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.190973    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-cs8xr"]
Nov 01 07:17:10.258964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.258931    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba3540-a6bb-464f-89b1-0288581d1d47-metrics-certs\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.259149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.258984    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3ba3540-a6bb-464f-89b1-0288581d1d47-webhook-cert\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.259149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.259016    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zq7b\" (UniqueName: \"kubernetes.io/projected/b3ba3540-a6bb-464f-89b1-0288581d1d47-kube-api-access-9zq7b\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.261907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.261882    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3ba3540-a6bb-464f-89b1-0288581d1d47-webhook-cert\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.262079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.261969    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba3540-a6bb-464f-89b1-0288581d1d47-metrics-certs\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.330433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.330353    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zq7b\" (UniqueName: \"kubernetes.io/projected/b3ba3540-a6bb-464f-89b1-0288581d1d47-kube-api-access-9zq7b\") pod \"maas-controller-64d475bcc9-cs8xr\" (UID: \"b3ba3540-a6bb-464f-89b1-0288581d1d47\") " pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.399205 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.399179    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:10.589222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.589143    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15667a56-6b92-4787-ac98-f633cadbe7df" path="/var/lib/kubelet/pods/15667a56-6b92-4787-ac98-f633cadbe7df/volumes"
Nov 01 07:17:10.798248 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:10.798222    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d475bcc9-cs8xr"]
Nov 01 07:17:10.800104 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:17:10.800079    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ba3540_a6bb_464f_89b1_0288581d1d47.slice/crio-ead36cbbabda0ea9e30862e161734d4a4406ad0289129078bb9bde73a9581540 WatchSource:0}: Error finding container ead36cbbabda0ea9e30862e161734d4a4406ad0289129078bb9bde73a9581540: Status 404 returned error can't find the container with id ead36cbbabda0ea9e30862e161734d4a4406ad0289129078bb9bde73a9581540
Nov 01 07:17:11.055832 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:11.055791    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 01 07:17:11.055997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:11.055852    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 01 07:17:11.547822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:11.547784    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-cs8xr" event={"ID":"b3ba3540-a6bb-464f-89b1-0288581d1d47","Type":"ContainerStarted","Data":"ead36cbbabda0ea9e30862e161734d4a4406ad0289129078bb9bde73a9581540"}
Nov 01 07:17:12.056152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:12.056111    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 01 07:17:12.056341 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:12.056164    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 01 07:17:12.554364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:12.554333    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d475bcc9-cs8xr" event={"ID":"b3ba3540-a6bb-464f-89b1-0288581d1d47","Type":"ContainerStarted","Data":"c429948c526d10b193033048796ca5ba49627cb3dd4485fe9b2feafabafb999c"}
Nov 01 07:17:12.554802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:12.554396    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:12.669634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:12.669580    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d475bcc9-cs8xr" podStartSLOduration=1.694588908 podStartE2EDuration="2.669564935s" totalImagesPullingTime="974.976027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:17:12.667364498 +0000 UTC m=+4044.235171328" watchObservedRunningTime="2026-11-01 07:17:12.669564935 +0000 UTC m=+4044.237371796"
Nov 01 07:17:13.056617 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:13.056579    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 01 07:17:13.056768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:13.056652    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 01 07:17:14.773762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:14.773719    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:14.774798 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:14.774778    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:14.775348 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:14.775329    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc"
Nov 01 07:17:23.561564 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:17:23.561532    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d475bcc9-cs8xr"
Nov 01 07:17:32.946995 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:17:32.946955    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:18:02.946966 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:18:02.946876    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:18:08.043933 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:18:08.043899    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"]
Nov 01 07:18:08.092872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:18:08.092835    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510896cc-trgkk"]
Nov 01 07:18:08.588595 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:18:08.588555    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a946347-3726-4a25-be00-53de646d2b94" path="/var/lib/kubelet/pods/9a946347-3726-4a25-be00-53de646d2b94/volumes"
Nov 01 07:18:20.043845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:18:20.043805    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"]
Nov 01 07:18:20.093018 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:18:20.092981    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec89464-6x9gn"]
Nov 01 07:18:20.589686 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:18:20.589650    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f881bc-4c71-4442-bb45-d7bff48c91f4" path="/var/lib/kubelet/pods/d9f881bc-4c71-4442-bb45-d7bff48c91f4/volumes"
Nov 01 07:18:32.946033 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:18:32.945991    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:19:02.947479 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:19:02.947425    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:19:11.078090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:11.078053    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:19:15.513913 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.513879    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"]
Nov 01 07:19:15.517494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.517478    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.571378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.571336    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:19:15.574175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.574141    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"]
Nov 01 07:19:15.589232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.589205    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lqt\" (UniqueName: \"kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.589358 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.589261    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.589442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.589423    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.620940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.620907    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:19:15.620940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.620907    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:19:15.621118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.621003    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:19:15.690404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.690371    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.690404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.690410    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lqt\" (UniqueName: \"kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.690671 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.690496    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.691179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.691158    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.693168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.693146    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.746737 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.746715    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lqt\" (UniqueName: \"kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt\") pod \"maas-api-revoke-sub-d576e68b-flrch\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:15.829120 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:15.829097    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:16.206355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:16.206330    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"]
Nov 01 07:19:16.208668 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:19:16.208641    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf72aa70_148c_4b18_9387_b5e4f215a88a.slice/crio-aa00af491a41265b607ba83a5a1361492a592ba0652d7b9a5981696ba246e000 WatchSource:0}: Error finding container aa00af491a41265b607ba83a5a1361492a592ba0652d7b9a5981696ba246e000: Status 404 returned error can't find the container with id aa00af491a41265b607ba83a5a1361492a592ba0652d7b9a5981696ba246e000
Nov 01 07:19:17.201356 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:17.201320    4468 generic.go:356] "Generic (PLEG): container finished" podID="bf72aa70-148c-4b18-9387-b5e4f215a88a" containerID="f2a3cb9e1c106a476ed20d28866004ba9464560d24a409afc8fb64c8c1d9a145" exitCode=0
Nov 01 07:19:17.201850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:17.201395    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch" event={"ID":"bf72aa70-148c-4b18-9387-b5e4f215a88a","Type":"ContainerDied","Data":"f2a3cb9e1c106a476ed20d28866004ba9464560d24a409afc8fb64c8c1d9a145"}
Nov 01 07:19:17.201850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:17.201449    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch" event={"ID":"bf72aa70-148c-4b18-9387-b5e4f215a88a","Type":"ContainerStarted","Data":"aa00af491a41265b607ba83a5a1361492a592ba0652d7b9a5981696ba246e000"}
Nov 01 07:19:18.339275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.339253    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:18.418280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.418247    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt\" (UniqueName: \"kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt\") pod \"bf72aa70-148c-4b18-9387-b5e4f215a88a\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") "
Nov 01 07:19:18.418461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.418333    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca\") pod \"bf72aa70-148c-4b18-9387-b5e4f215a88a\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") "
Nov 01 07:19:18.418461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.418362    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca\") pod \"bf72aa70-148c-4b18-9387-b5e4f215a88a\" (UID: \"bf72aa70-148c-4b18-9387-b5e4f215a88a\") "
Nov 01 07:19:18.418741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.418716    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca" pod "bf72aa70-148c-4b18-9387-b5e4f215a88a" (UID: "bf72aa70-148c-4b18-9387-b5e4f215a88a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:19:18.420440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.420422    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt" pod "bf72aa70-148c-4b18-9387-b5e4f215a88a" (UID: "bf72aa70-148c-4b18-9387-b5e4f215a88a"). InnerVolumeSpecName "kube-api-access-66lqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:19:18.420440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.420430    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca" pod "bf72aa70-148c-4b18-9387-b5e4f215a88a" (UID: "bf72aa70-148c-4b18-9387-b5e4f215a88a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:19:18.519882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.519820    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66lqt\" (UniqueName: \"kubernetes.io/projected/bf72aa70-148c-4b18-9387-b5e4f215a88a-kube-api-access-66lqt\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:18.519882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.519844    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:18.519882 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:18.519855    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72aa70-148c-4b18-9387-b5e4f215a88a-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:19:19.212283 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:19.212253    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"
Nov 01 07:19:19.212456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:19.212254    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch" event={"ID":"bf72aa70-148c-4b18-9387-b5e4f215a88a","Type":"ContainerDied","Data":"aa00af491a41265b607ba83a5a1361492a592ba0652d7b9a5981696ba246e000"}
Nov 01 07:19:19.212456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:19.212387    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa00af491a41265b607ba83a5a1361492a592ba0652d7b9a5981696ba246e000"
Nov 01 07:19:30.683267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:30.683181    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:19:32.948321 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:19:32.948278    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:19:43.487516 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:43.487479    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:19:52.689150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:52.689118    4468 scope.go:122] "RemoveContainer" containerID="6ba6df317fbf966da7bc49949ba7a72de9571039000c9018fa68c502c8e256ed"
Nov 01 07:19:52.697669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:19:52.697633    4468 scope.go:122] "RemoveContainer" containerID="bf90c5160ca4c15d12c91bca207899d4394189751d6e7a44090f9e8350013cb8"
Nov 01 07:20:00.819492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:00.818601    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:20:00.819492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:00.818683    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:20:02.946867 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:20:02.946830    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:20:05.907036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:05.906993    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:20:13.749192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.749150    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"]
Nov 01 07:20:13.749701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.749625    4468 cpu_manager.go:414] "containerMap: removing container" podUID="bf72aa70-148c-4b18-9387-b5e4f215a88a" containerName="revoke-keys"
Nov 01 07:20:13.749701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.749638    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf72aa70-148c-4b18-9387-b5e4f215a88a" containerName="revoke-keys"
Nov 01 07:20:13.749701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.749695    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf72aa70-148c-4b18-9387-b5e4f215a88a" containerName="revoke-keys"
Nov 01 07:20:13.752951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.752935    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.809635 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.809600    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"]
Nov 01 07:20:13.843736 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.843698    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.843925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.843847    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.843985 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.843962    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9gs5\" (UniqueName: \"kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.856532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.856498    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:20:13.856532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.856500    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:20:13.856722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.856531    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:20:13.856722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.856576    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:20:13.945405 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.945358    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.945652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.945453    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9gs5\" (UniqueName: \"kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.945652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.945544    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.946061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.946040    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:13.948056 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:13.948031    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:14.001961 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:14.001883    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9gs5\" (UniqueName: \"kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5\") pod \"maas-api-revoke-sub-e85756a3-mvnhm\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:14.066048 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:14.066014    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:14.440809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:14.440782    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"]
Nov 01 07:20:14.443082 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:20:14.443048    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c761959_c16a_4ba6_9d14_b66f5ec10ca3.slice/crio-9a94865dcca82d7cb7e88171dfe36ce14999e033c63e049c70aa17deea6304ee WatchSource:0}: Error finding container 9a94865dcca82d7cb7e88171dfe36ce14999e033c63e049c70aa17deea6304ee: Status 404 returned error can't find the container with id 9a94865dcca82d7cb7e88171dfe36ce14999e033c63e049c70aa17deea6304ee
Nov 01 07:20:14.570335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:14.570299    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm" event={"ID":"7c761959-c16a-4ba6-9d14-b66f5ec10ca3","Type":"ContainerStarted","Data":"9a94865dcca82d7cb7e88171dfe36ce14999e033c63e049c70aa17deea6304ee"}
Nov 01 07:20:15.576759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:15.576725    4468 generic.go:356] "Generic (PLEG): container finished" podID="7c761959-c16a-4ba6-9d14-b66f5ec10ca3" containerID="5c064786a3d0077d50f54d670aaf143ea1dcda4761df63d1715daa21421f0445" exitCode=0
Nov 01 07:20:15.577178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:15.576810    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm" event={"ID":"7c761959-c16a-4ba6-9d14-b66f5ec10ca3","Type":"ContainerDied","Data":"5c064786a3d0077d50f54d670aaf143ea1dcda4761df63d1715daa21421f0445"}
Nov 01 07:20:16.722750 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.722721    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:16.774527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.774493    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5\" (UniqueName: \"kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5\") pod \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") "
Nov 01 07:20:16.774718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.774617    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca\") pod \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") "
Nov 01 07:20:16.774718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.774648    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca\") pod \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\" (UID: \"7c761959-c16a-4ba6-9d14-b66f5ec10ca3\") "
Nov 01 07:20:16.774986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.774966    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca" pod "7c761959-c16a-4ba6-9d14-b66f5ec10ca3" (UID: "7c761959-c16a-4ba6-9d14-b66f5ec10ca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:20:16.776920 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.776893    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca" pod "7c761959-c16a-4ba6-9d14-b66f5ec10ca3" (UID: "7c761959-c16a-4ba6-9d14-b66f5ec10ca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:20:16.777036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.776933    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5" pod "7c761959-c16a-4ba6-9d14-b66f5ec10ca3" (UID: "7c761959-c16a-4ba6-9d14-b66f5ec10ca3"). InnerVolumeSpecName "kube-api-access-j9gs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:20:16.876479 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.876383    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:16.876479 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.876415    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:16.876479 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:16.876424    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9gs5\" (UniqueName: \"kubernetes.io/projected/7c761959-c16a-4ba6-9d14-b66f5ec10ca3-kube-api-access-j9gs5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:17.589582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:17.589544    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm" event={"ID":"7c761959-c16a-4ba6-9d14-b66f5ec10ca3","Type":"ContainerDied","Data":"9a94865dcca82d7cb7e88171dfe36ce14999e033c63e049c70aa17deea6304ee"}
Nov 01 07:20:17.589766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:17.589602    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a94865dcca82d7cb7e88171dfe36ce14999e033c63e049c70aa17deea6304ee"
Nov 01 07:20:17.589766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:17.589557    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"
Nov 01 07:20:24.752565 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.752515    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"]
Nov 01 07:20:24.753062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.752957    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7c761959-c16a-4ba6-9d14-b66f5ec10ca3" containerName="revoke-keys"
Nov 01 07:20:24.753062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.752968    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c761959-c16a-4ba6-9d14-b66f5ec10ca3" containerName="revoke-keys"
Nov 01 07:20:24.753062 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.753044    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c761959-c16a-4ba6-9d14-b66f5ec10ca3" containerName="revoke-keys"
Nov 01 07:20:24.761422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.761382    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.812221 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.812181    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"]
Nov 01 07:20:24.813053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.813023    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:20:24.859214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.859178    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:20:24.859214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.859202    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:20:24.859214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.859202    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:20:24.862746 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.862723    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.862805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.862786    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97tq\" (UniqueName: \"kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.863002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.862982    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.963653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.963595    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.963865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.963704    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97tq\" (UniqueName: \"kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.963865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.963820    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.964607 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.964585    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:24.965987 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:24.965970    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:25.020415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:25.020337    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97tq\" (UniqueName: \"kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq\") pod \"maas-api-revoke-sub-cbe4a15f-ds9z6\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:25.074159 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:25.074112    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:25.451874 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:25.451846    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"]
Nov 01 07:20:25.453712 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:20:25.453681    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35f0354_6517_462d_9248_6df4dcaa4e76.slice/crio-b4b65995265aff625ced8ecff60ad501b9d6aadd78f3f0d0134ea767acb5e3ca WatchSource:0}: Error finding container b4b65995265aff625ced8ecff60ad501b9d6aadd78f3f0d0134ea767acb5e3ca: Status 404 returned error can't find the container with id b4b65995265aff625ced8ecff60ad501b9d6aadd78f3f0d0134ea767acb5e3ca
Nov 01 07:20:25.627172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:25.627141    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6" event={"ID":"c35f0354-6517-462d-9248-6df4dcaa4e76","Type":"ContainerStarted","Data":"b4b65995265aff625ced8ecff60ad501b9d6aadd78f3f0d0134ea767acb5e3ca"}
Nov 01 07:20:26.633559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:26.633520    4468 generic.go:356] "Generic (PLEG): container finished" podID="c35f0354-6517-462d-9248-6df4dcaa4e76" containerID="7592283ec4f11fcb4de4967b7244acf9bb72fb2a6c5ea204c58612737c201652" exitCode=0
Nov 01 07:20:26.634011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:26.633602    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6" event={"ID":"c35f0354-6517-462d-9248-6df4dcaa4e76","Type":"ContainerDied","Data":"7592283ec4f11fcb4de4967b7244acf9bb72fb2a6c5ea204c58612737c201652"}
Nov 01 07:20:27.777714 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.777691    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:27.893646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.893569    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq\" (UniqueName: \"kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq\") pod \"c35f0354-6517-462d-9248-6df4dcaa4e76\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") "
Nov 01 07:20:27.893803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.893656    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca\") pod \"c35f0354-6517-462d-9248-6df4dcaa4e76\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") "
Nov 01 07:20:27.893803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.893697    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca\") pod \"c35f0354-6517-462d-9248-6df4dcaa4e76\" (UID: \"c35f0354-6517-462d-9248-6df4dcaa4e76\") "
Nov 01 07:20:27.894109 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.894074    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca" pod "c35f0354-6517-462d-9248-6df4dcaa4e76" (UID: "c35f0354-6517-462d-9248-6df4dcaa4e76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:20:27.895833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.895811    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca" pod "c35f0354-6517-462d-9248-6df4dcaa4e76" (UID: "c35f0354-6517-462d-9248-6df4dcaa4e76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:20:27.896041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.896017    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq" pod "c35f0354-6517-462d-9248-6df4dcaa4e76" (UID: "c35f0354-6517-462d-9248-6df4dcaa4e76"). InnerVolumeSpecName "kube-api-access-p97tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:20:27.995244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.995202    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:27.995244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.995239    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c35f0354-6517-462d-9248-6df4dcaa4e76-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:27.995461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:27.995255    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97tq\" (UniqueName: \"kubernetes.io/projected/c35f0354-6517-462d-9248-6df4dcaa4e76-kube-api-access-p97tq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:28.644714 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:28.644685    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"
Nov 01 07:20:28.644894 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:28.644721    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6" event={"ID":"c35f0354-6517-462d-9248-6df4dcaa4e76","Type":"ContainerDied","Data":"b4b65995265aff625ced8ecff60ad501b9d6aadd78f3f0d0134ea767acb5e3ca"}
Nov 01 07:20:28.644894 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:28.644773    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b65995265aff625ced8ecff60ad501b9d6aadd78f3f0d0134ea767acb5e3ca"
Nov 01 07:20:28.682162 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:28.682128    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:20:32.947287 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:20:32.947245    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:20:45.877867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.877835    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"]
Nov 01 07:20:45.878276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.878258    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c35f0354-6517-462d-9248-6df4dcaa4e76" containerName="revoke-keys"
Nov 01 07:20:45.878276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.878269    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35f0354-6517-462d-9248-6df4dcaa4e76" containerName="revoke-keys"
Nov 01 07:20:45.878367 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.878357    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35f0354-6517-462d-9248-6df4dcaa4e76" containerName="revoke-keys"
Nov 01 07:20:45.881576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.881558    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:45.911818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.911789    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:45.911947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.911859    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:45.911947 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.911941    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx49x\" (UniqueName: \"kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:45.933493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.933452    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:20:45.933599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.933516    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:20:45.933599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.933561    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:20:45.985699 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:45.985672    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"]
Nov 01 07:20:46.012905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.012878    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.013026 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.012950    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.013026 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.013000    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx49x\" (UniqueName: \"kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.013558 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.013539    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.015291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.015273    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.033812 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.033790    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:20:46.069238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.069215    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx49x\" (UniqueName: \"kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x\") pod \"maas-api-revoke-sub-bbabb089-rqzzw\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.193547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.193477    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:46.563321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.563295    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"]
Nov 01 07:20:46.564590 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:20:46.564568    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22b6a2e_a553_4ae6_bbb5_4cff8d3ed9d9.slice/crio-53f0bcb8e6342ceb51fdbb62ee3af2747d470dc431cae0c2581a4600a741756a WatchSource:0}: Error finding container 53f0bcb8e6342ceb51fdbb62ee3af2747d470dc431cae0c2581a4600a741756a: Status 404 returned error can't find the container with id 53f0bcb8e6342ceb51fdbb62ee3af2747d470dc431cae0c2581a4600a741756a
Nov 01 07:20:46.753894 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:46.753855    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw" event={"ID":"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9","Type":"ContainerStarted","Data":"53f0bcb8e6342ceb51fdbb62ee3af2747d470dc431cae0c2581a4600a741756a"}
Nov 01 07:20:47.760327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:47.760249    4468 generic.go:356] "Generic (PLEG): container finished" podID="f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" containerID="191fb4dafbeb4bf59af0ee6230dd387a7ebc01f0f4db95f77503558550a59657" exitCode=0
Nov 01 07:20:47.760327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:47.760282    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw" event={"ID":"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9","Type":"ContainerDied","Data":"191fb4dafbeb4bf59af0ee6230dd387a7ebc01f0f4db95f77503558550a59657"}
Nov 01 07:20:48.895782 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.895751    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:48.939011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.938985    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca\") pod \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") "
Nov 01 07:20:48.939122 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.939080    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x\" (UniqueName: \"kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x\") pod \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") "
Nov 01 07:20:48.939158 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.939123    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca\") pod \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\" (UID: \"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9\") "
Nov 01 07:20:48.939488 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.939447    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca" pod "f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" (UID: "f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:20:48.939615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.939601    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:48.941021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.941004    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca" pod "f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" (UID: "f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:20:48.941152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:48.941138    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x" pod "f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" (UID: "f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9"). InnerVolumeSpecName "kube-api-access-wx49x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:20:49.041048 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:49.040980    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx49x\" (UniqueName: \"kubernetes.io/projected/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-kube-api-access-wx49x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:49.041048 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:49.041005    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:20:49.772774 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:49.772733    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw" event={"ID":"f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9","Type":"ContainerDied","Data":"53f0bcb8e6342ceb51fdbb62ee3af2747d470dc431cae0c2581a4600a741756a"}
Nov 01 07:20:49.772988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:49.772790    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"
Nov 01 07:20:49.772988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:49.772799    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f0bcb8e6342ceb51fdbb62ee3af2747d470dc431cae0c2581a4600a741756a"
Nov 01 07:20:56.823291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.823252    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"]
Nov 01 07:20:56.823845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.823726    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" containerName="revoke-keys"
Nov 01 07:20:56.823845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.823738    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" containerName="revoke-keys"
Nov 01 07:20:56.823845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.823831    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" containerName="revoke-keys"
Nov 01 07:20:56.826935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.826914    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:56.881607 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.881575    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:20:56.914193 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.914158    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:56.914388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.914217    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:56.914442 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.914374    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjw5\" (UniqueName: \"kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:56.931225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.931194    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:20:56.931225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.931207    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:20:56.931409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.931253    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:20:56.933499 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:56.933478    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"]
Nov 01 07:20:57.015275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.015243    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.015490 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.015288    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.015490 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.015358    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjw5\" (UniqueName: \"kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.016172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.016140    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.018015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.017993    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.071991 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.071954    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjw5\" (UniqueName: \"kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5\") pod \"maas-api-revoke-sub-a1346d1b-lnj6l\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.139659 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.139565    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:20:57.508259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.508233    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"]
Nov 01 07:20:57.509814 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:20:57.509784    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be6894e_4348_4199_a3dd_c59b83bee976.slice/crio-bd7b16d9991bd787b8c590c7ada634796afae29b69d11b75145ab96a5c5e816d WatchSource:0}: Error finding container bd7b16d9991bd787b8c590c7ada634796afae29b69d11b75145ab96a5c5e816d: Status 404 returned error can't find the container with id bd7b16d9991bd787b8c590c7ada634796afae29b69d11b75145ab96a5c5e816d
Nov 01 07:20:57.814754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:57.814726    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l" event={"ID":"6be6894e-4348-4199-a3dd-c59b83bee976","Type":"ContainerStarted","Data":"bd7b16d9991bd787b8c590c7ada634796afae29b69d11b75145ab96a5c5e816d"}
Nov 01 07:20:58.821416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:58.821378    4468 generic.go:356] "Generic (PLEG): container finished" podID="6be6894e-4348-4199-a3dd-c59b83bee976" containerID="ed8cbaae077420f278db8e63f1196e66744c924aca8728022802c1cfdc37adca" exitCode=0
Nov 01 07:20:58.821859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:58.821440    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l" event={"ID":"6be6894e-4348-4199-a3dd-c59b83bee976","Type":"ContainerDied","Data":"ed8cbaae077420f278db8e63f1196e66744c924aca8728022802c1cfdc37adca"}
Nov 01 07:20:59.989187 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:20:59.989157    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:21:00.146245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.146149    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca\") pod \"6be6894e-4348-4199-a3dd-c59b83bee976\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") "
Nov 01 07:21:00.146245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.146227    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca\") pod \"6be6894e-4348-4199-a3dd-c59b83bee976\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") "
Nov 01 07:21:00.146496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.146315    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5\" (UniqueName: \"kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5\") pod \"6be6894e-4348-4199-a3dd-c59b83bee976\" (UID: \"6be6894e-4348-4199-a3dd-c59b83bee976\") "
Nov 01 07:21:00.146662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.146631    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca" pod "6be6894e-4348-4199-a3dd-c59b83bee976" (UID: "6be6894e-4348-4199-a3dd-c59b83bee976"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:21:00.148532 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.148503    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca" pod "6be6894e-4348-4199-a3dd-c59b83bee976" (UID: "6be6894e-4348-4199-a3dd-c59b83bee976"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:21:00.148977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.148958    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5" pod "6be6894e-4348-4199-a3dd-c59b83bee976" (UID: "6be6894e-4348-4199-a3dd-c59b83bee976"). InnerVolumeSpecName "kube-api-access-wgjw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:21:00.247667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.247627    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgjw5\" (UniqueName: \"kubernetes.io/projected/6be6894e-4348-4199-a3dd-c59b83bee976-kube-api-access-wgjw5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:00.247667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.247658    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:00.247667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.247669    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be6894e-4348-4199-a3dd-c59b83bee976-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:00.866551 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.862915    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l" event={"ID":"6be6894e-4348-4199-a3dd-c59b83bee976","Type":"ContainerDied","Data":"bd7b16d9991bd787b8c590c7ada634796afae29b69d11b75145ab96a5c5e816d"}
Nov 01 07:21:00.866551 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.862980    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7b16d9991bd787b8c590c7ada634796afae29b69d11b75145ab96a5c5e816d"
Nov 01 07:21:00.866551 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:00.863071    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"
Nov 01 07:21:02.947524 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:21:02.947485    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:21:07.830029 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.829986    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"]
Nov 01 07:21:07.830625 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.830604    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6be6894e-4348-4199-a3dd-c59b83bee976" containerName="revoke-keys"
Nov 01 07:21:07.830722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.830627    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be6894e-4348-4199-a3dd-c59b83bee976" containerName="revoke-keys"
Nov 01 07:21:07.830799 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.830786    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be6894e-4348-4199-a3dd-c59b83bee976" containerName="revoke-keys"
Nov 01 07:21:07.833808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.833787    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.852178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.852151    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.852302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.852222    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.852375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.852327    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxd5\" (UniqueName: \"kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.889605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.889567    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"]
Nov 01 07:21:07.936386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.936357    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:21:07.936386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.936360    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:21:07.936618 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.936358    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:21:07.936618 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.936361    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:21:07.953004 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.952972    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.953091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.953064    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxd5\" (UniqueName: \"kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.953136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.953101    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.953730 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.953704    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:07.955600 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:07.955572    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:08.009305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:08.009279    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxd5\" (UniqueName: \"kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5\") pod \"maas-api-revoke-sub-bb27aa15-zvlh5\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:08.146352 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:08.146274    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:08.520627 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:08.520597    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"]
Nov 01 07:21:08.522813 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:21:08.522777    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a64407_8623_46ed_9e57_efa8509bb9e0.slice/crio-b587d8c2c3e92a924e7a3965c5f433071dc3e7032e477279619ff418adfab99f WatchSource:0}: Error finding container b587d8c2c3e92a924e7a3965c5f433071dc3e7032e477279619ff418adfab99f: Status 404 returned error can't find the container with id b587d8c2c3e92a924e7a3965c5f433071dc3e7032e477279619ff418adfab99f
Nov 01 07:21:08.909128 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:08.909090    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5" event={"ID":"77a64407-8623-46ed-9e57-efa8509bb9e0","Type":"ContainerStarted","Data":"eeb9f1549151ec37ef7a85dd4c30850d85496b758c01ef5bbf9ea4457674af2c"}
Nov 01 07:21:08.909538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:08.909150    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5" event={"ID":"77a64407-8623-46ed-9e57-efa8509bb9e0","Type":"ContainerStarted","Data":"b587d8c2c3e92a924e7a3965c5f433071dc3e7032e477279619ff418adfab99f"}
Nov 01 07:21:09.022890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:09.022836    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5" podStartSLOduration=2.022821781 podStartE2EDuration="2.022821781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:09.020743256 +0000 UTC m=+4280.588550085" watchObservedRunningTime="2026-11-01 07:21:09.022821781 +0000 UTC m=+4280.590628609"
Nov 01 07:21:09.915129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:09.915092    4468 generic.go:356] "Generic (PLEG): container finished" podID="77a64407-8623-46ed-9e57-efa8509bb9e0" containerID="eeb9f1549151ec37ef7a85dd4c30850d85496b758c01ef5bbf9ea4457674af2c" exitCode=0
Nov 01 07:21:09.915561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:09.915172    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5" event={"ID":"77a64407-8623-46ed-9e57-efa8509bb9e0","Type":"ContainerDied","Data":"eeb9f1549151ec37ef7a85dd4c30850d85496b758c01ef5bbf9ea4457674af2c"}
Nov 01 07:21:11.090299 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.090258    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:11.189845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.189746    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca\") pod \"77a64407-8623-46ed-9e57-efa8509bb9e0\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") "
Nov 01 07:21:11.189845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.189815    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca\") pod \"77a64407-8623-46ed-9e57-efa8509bb9e0\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") "
Nov 01 07:21:11.190102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.189851    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5\" (UniqueName: \"kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5\") pod \"77a64407-8623-46ed-9e57-efa8509bb9e0\" (UID: \"77a64407-8623-46ed-9e57-efa8509bb9e0\") "
Nov 01 07:21:11.190207 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.190175    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca" pod "77a64407-8623-46ed-9e57-efa8509bb9e0" (UID: "77a64407-8623-46ed-9e57-efa8509bb9e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:21:11.193149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.193022    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca" pod "77a64407-8623-46ed-9e57-efa8509bb9e0" (UID: "77a64407-8623-46ed-9e57-efa8509bb9e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:21:11.193149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.193068    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5" pod "77a64407-8623-46ed-9e57-efa8509bb9e0" (UID: "77a64407-8623-46ed-9e57-efa8509bb9e0"). InnerVolumeSpecName "kube-api-access-jjxd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:21:11.291047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.291003    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:11.291047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.291047    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a64407-8623-46ed-9e57-efa8509bb9e0-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:11.291342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.291062    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxd5\" (UniqueName: \"kubernetes.io/projected/77a64407-8623-46ed-9e57-efa8509bb9e0-kube-api-access-jjxd5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:11.968383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.968335    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5" event={"ID":"77a64407-8623-46ed-9e57-efa8509bb9e0","Type":"ContainerDied","Data":"b587d8c2c3e92a924e7a3965c5f433071dc3e7032e477279619ff418adfab99f"}
Nov 01 07:21:11.968734 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.968694    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b587d8c2c3e92a924e7a3965c5f433071dc3e7032e477279619ff418adfab99f"
Nov 01 07:21:11.969068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:11.969052    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"
Nov 01 07:21:18.819812 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.819772    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"]
Nov 01 07:21:18.820438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.820417    4468 cpu_manager.go:414] "containerMap: removing container" podUID="77a64407-8623-46ed-9e57-efa8509bb9e0" containerName="revoke-keys"
Nov 01 07:21:18.820573 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.820441    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a64407-8623-46ed-9e57-efa8509bb9e0" containerName="revoke-keys"
Nov 01 07:21:18.820630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.820588    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a64407-8623-46ed-9e57-efa8509bb9e0" containerName="revoke-keys"
Nov 01 07:21:18.823857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.823836    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.880399 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.880362    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"]
Nov 01 07:21:18.885013 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.884988    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.885160 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.885063    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.885160 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.885152    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.926974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.926933    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:21:18.926974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.926949    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:21:18.926974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.926954    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:21:18.927214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.927007    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:21:18.986291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.986253    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.986543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.986305    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.986543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.986337    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.987021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.986996    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:18.988930 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:18.988909    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:19.042479 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:19.042445    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf\") pod \"maas-api-revoke-sub-2ff87d5e-kccjc\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:19.136372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:19.136309    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:19.507518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:19.507495    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"]
Nov 01 07:21:19.509359 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:21:19.509331    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7e0379_bd90_4161_a50d_7f8558adb485.slice/crio-7c16188c37cff7483a98054eb34aed583d9b95724b86ffa7a4412a389f8dff49 WatchSource:0}: Error finding container 7c16188c37cff7483a98054eb34aed583d9b95724b86ffa7a4412a389f8dff49: Status 404 returned error can't find the container with id 7c16188c37cff7483a98054eb34aed583d9b95724b86ffa7a4412a389f8dff49
Nov 01 07:21:20.010219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:20.010189    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc" event={"ID":"1b7e0379-bd90-4161-a50d-7f8558adb485","Type":"ContainerStarted","Data":"46479d5e8b95013d78088ea58510cb42f304d7cf73aa8dfddaede2272b332e21"}
Nov 01 07:21:20.010710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:20.010248    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc" event={"ID":"1b7e0379-bd90-4161-a50d-7f8558adb485","Type":"ContainerStarted","Data":"7c16188c37cff7483a98054eb34aed583d9b95724b86ffa7a4412a389f8dff49"}
Nov 01 07:21:21.016655 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:21.016623    4468 generic.go:356] "Generic (PLEG): container finished" podID="1b7e0379-bd90-4161-a50d-7f8558adb485" containerID="46479d5e8b95013d78088ea58510cb42f304d7cf73aa8dfddaede2272b332e21" exitCode=0
Nov 01 07:21:21.017036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:21.016681    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc" event={"ID":"1b7e0379-bd90-4161-a50d-7f8558adb485","Type":"ContainerDied","Data":"46479d5e8b95013d78088ea58510cb42f304d7cf73aa8dfddaede2272b332e21"}
Nov 01 07:21:22.190645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.190608    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:22.331492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.326796    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf\") pod \"1b7e0379-bd90-4161-a50d-7f8558adb485\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") "
Nov 01 07:21:22.331492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.326872    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca\") pod \"1b7e0379-bd90-4161-a50d-7f8558adb485\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") "
Nov 01 07:21:22.331492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.326991    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca\") pod \"1b7e0379-bd90-4161-a50d-7f8558adb485\" (UID: \"1b7e0379-bd90-4161-a50d-7f8558adb485\") "
Nov 01 07:21:22.371793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.363883    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf" pod "1b7e0379-bd90-4161-a50d-7f8558adb485" (UID: "1b7e0379-bd90-4161-a50d-7f8558adb485"). InnerVolumeSpecName "kube-api-access-sz9pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:21:22.371793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.371646    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca" pod "1b7e0379-bd90-4161-a50d-7f8558adb485" (UID: "1b7e0379-bd90-4161-a50d-7f8558adb485"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:21:22.371793 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.371762    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca" pod "1b7e0379-bd90-4161-a50d-7f8558adb485" (UID: "1b7e0379-bd90-4161-a50d-7f8558adb485"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:21:22.431495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.431161    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/1b7e0379-bd90-4161-a50d-7f8558adb485-kube-api-access-sz9pf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:22.431495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.431203    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:22.431495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:22.431222    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7e0379-bd90-4161-a50d-7f8558adb485-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:23.039264 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:23.039177    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc" event={"ID":"1b7e0379-bd90-4161-a50d-7f8558adb485","Type":"ContainerDied","Data":"7c16188c37cff7483a98054eb34aed583d9b95724b86ffa7a4412a389f8dff49"}
Nov 01 07:21:23.053491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:23.039526    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c16188c37cff7483a98054eb34aed583d9b95724b86ffa7a4412a389f8dff49"
Nov 01 07:21:23.053877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:23.053859    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"
Nov 01 07:21:29.871495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.871424    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"]
Nov 01 07:21:29.872139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.872116    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1b7e0379-bd90-4161-a50d-7f8558adb485" containerName="revoke-keys"
Nov 01 07:21:29.872228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.872143    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7e0379-bd90-4161-a50d-7f8558adb485" containerName="revoke-keys"
Nov 01 07:21:29.872291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.872281    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7e0379-bd90-4161-a50d-7f8558adb485" containerName="revoke-keys"
Nov 01 07:21:29.875520 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.875500    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:29.928407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.928344    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:21:29.928607 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.928414    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:21:29.977694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.977661    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:21:29.977863 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.977767    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:21:29.980158 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:29.980140    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"]
Nov 01 07:21:30.043135 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.043105    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.043312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.043152    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.043388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.043327    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbkw\" (UniqueName: \"kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.144521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.144410    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.144521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.144481    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.144767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.144567    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbkw\" (UniqueName: \"kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.145153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.145126    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.147019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.146999    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.200677 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.200655    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbkw\" (UniqueName: \"kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw\") pod \"maas-api-revoke-sub-d4f6480b-6gl9n\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.487656 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.487580    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:30.857752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:30.857726    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"]
Nov 01 07:21:30.859570 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:21:30.859548    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dda460f_8a3d_43a6_a907_523d9262adbd.slice/crio-38760e9288c5ac0cb563064d19f6c46e546fce7a1c6f5f875fc660fc8c04b5af WatchSource:0}: Error finding container 38760e9288c5ac0cb563064d19f6c46e546fce7a1c6f5f875fc660fc8c04b5af: Status 404 returned error can't find the container with id 38760e9288c5ac0cb563064d19f6c46e546fce7a1c6f5f875fc660fc8c04b5af
Nov 01 07:21:31.080103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:31.080070    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n" event={"ID":"2dda460f-8a3d-43a6-a907-523d9262adbd","Type":"ContainerStarted","Data":"38760e9288c5ac0cb563064d19f6c46e546fce7a1c6f5f875fc660fc8c04b5af"}
Nov 01 07:21:32.087584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:32.087547    4468 generic.go:356] "Generic (PLEG): container finished" podID="2dda460f-8a3d-43a6-a907-523d9262adbd" containerID="72b37b634b893f939a513d58c429acdc12bfcdf6602db545701e3cf8eed6b8d5" exitCode=0
Nov 01 07:21:32.088047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:32.087634    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n" event={"ID":"2dda460f-8a3d-43a6-a907-523d9262adbd","Type":"ContainerDied","Data":"72b37b634b893f939a513d58c429acdc12bfcdf6602db545701e3cf8eed6b8d5"}
Nov 01 07:21:32.960904 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:21:32.960859    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:21:33.248669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.248640    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:33.382238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.381226    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca\") pod \"2dda460f-8a3d-43a6-a907-523d9262adbd\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") "
Nov 01 07:21:33.382238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.381301    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca\") pod \"2dda460f-8a3d-43a6-a907-523d9262adbd\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") "
Nov 01 07:21:33.382238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.381418    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw\" (UniqueName: \"kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw\") pod \"2dda460f-8a3d-43a6-a907-523d9262adbd\" (UID: \"2dda460f-8a3d-43a6-a907-523d9262adbd\") "
Nov 01 07:21:33.384265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.383990    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca" pod "2dda460f-8a3d-43a6-a907-523d9262adbd" (UID: "2dda460f-8a3d-43a6-a907-523d9262adbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:21:33.387073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.387044    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw" pod "2dda460f-8a3d-43a6-a907-523d9262adbd" (UID: "2dda460f-8a3d-43a6-a907-523d9262adbd"). InnerVolumeSpecName "kube-api-access-5sbkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:21:33.388198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.388175    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca" pod "2dda460f-8a3d-43a6-a907-523d9262adbd" (UID: "2dda460f-8a3d-43a6-a907-523d9262adbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:21:33.483136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.483042    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:33.483136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.483093    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda460f-8a3d-43a6-a907-523d9262adbd-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:33.483136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:33.483110    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbkw\" (UniqueName: \"kubernetes.io/projected/2dda460f-8a3d-43a6-a907-523d9262adbd-kube-api-access-5sbkw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:34.102388 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:34.102362    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"
Nov 01 07:21:34.102662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:34.102393    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n" event={"ID":"2dda460f-8a3d-43a6-a907-523d9262adbd","Type":"ContainerDied","Data":"38760e9288c5ac0cb563064d19f6c46e546fce7a1c6f5f875fc660fc8c04b5af"}
Nov 01 07:21:34.102662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:34.102446    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38760e9288c5ac0cb563064d19f6c46e546fce7a1c6f5f875fc660fc8c04b5af"
Nov 01 07:21:40.621285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.621247    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"]
Nov 01 07:21:40.621744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.621693    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2dda460f-8a3d-43a6-a907-523d9262adbd" containerName="revoke-keys"
Nov 01 07:21:40.621744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.621705    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dda460f-8a3d-43a6-a907-523d9262adbd" containerName="revoke-keys"
Nov 01 07:21:40.621833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.621782    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dda460f-8a3d-43a6-a907-523d9262adbd" containerName="revoke-keys"
Nov 01 07:21:40.625003 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.624985    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.661014 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.660972    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtv6v\" (UniqueName: \"kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.661192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.661049    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.661192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.661127    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.728330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.728286    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:21:40.728330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.728285    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:21:40.728330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.728301    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:21:40.728693 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.728342    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:21:40.730867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.730845    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"]
Nov 01 07:21:40.762761 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.762729    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtv6v\" (UniqueName: \"kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.762949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.762788    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.762949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.762815    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.763541 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.763514    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.765524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.765506    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.819117 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.819074    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtv6v\" (UniqueName: \"kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v\") pod \"maas-api-revoke-sub-00e18343-lrfhp\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:40.938010 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:40.937915    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:41.308710 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:41.308682    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"]
Nov 01 07:21:41.309839 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:21:41.309816    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4339d562_2402_4478_99bd_a05fb1c396c8.slice/crio-08e6324f45f448cbbd0b488115227ff7de01c784cf3b7ab8eee9d7d9d8925d2b WatchSource:0}: Error finding container 08e6324f45f448cbbd0b488115227ff7de01c784cf3b7ab8eee9d7d9d8925d2b: Status 404 returned error can't find the container with id 08e6324f45f448cbbd0b488115227ff7de01c784cf3b7ab8eee9d7d9d8925d2b
Nov 01 07:21:42.145481 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:42.145433    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp" event={"ID":"4339d562-2402-4478-99bd-a05fb1c396c8","Type":"ContainerStarted","Data":"0772cf96d1737472b266199c44b38a77222adf21aac5cec528f29a84eb2108d2"}
Nov 01 07:21:42.145968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:42.145510    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp" event={"ID":"4339d562-2402-4478-99bd-a05fb1c396c8","Type":"ContainerStarted","Data":"08e6324f45f448cbbd0b488115227ff7de01c784cf3b7ab8eee9d7d9d8925d2b"}
Nov 01 07:21:42.258116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:42.258050    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp" podStartSLOduration=2.258031811 podStartE2EDuration="2.258031811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:42.257198997 +0000 UTC m=+4313.825005827" watchObservedRunningTime="2026-11-01 07:21:42.258031811 +0000 UTC m=+4313.825838643"
Nov 01 07:21:43.151936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:43.151903    4468 generic.go:356] "Generic (PLEG): container finished" podID="4339d562-2402-4478-99bd-a05fb1c396c8" containerID="0772cf96d1737472b266199c44b38a77222adf21aac5cec528f29a84eb2108d2" exitCode=0
Nov 01 07:21:43.152354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:43.151957    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp" event={"ID":"4339d562-2402-4478-99bd-a05fb1c396c8","Type":"ContainerDied","Data":"0772cf96d1737472b266199c44b38a77222adf21aac5cec528f29a84eb2108d2"}
Nov 01 07:21:44.318501 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.318451    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:44.409821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.408802    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v\" (UniqueName: \"kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v\") pod \"4339d562-2402-4478-99bd-a05fb1c396c8\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") "
Nov 01 07:21:44.409821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.408871    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca\") pod \"4339d562-2402-4478-99bd-a05fb1c396c8\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") "
Nov 01 07:21:44.409821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.408972    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca\") pod \"4339d562-2402-4478-99bd-a05fb1c396c8\" (UID: \"4339d562-2402-4478-99bd-a05fb1c396c8\") "
Nov 01 07:21:44.415954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.414806    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca" pod "4339d562-2402-4478-99bd-a05fb1c396c8" (UID: "4339d562-2402-4478-99bd-a05fb1c396c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:21:44.452488 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.448633    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v" pod "4339d562-2402-4478-99bd-a05fb1c396c8" (UID: "4339d562-2402-4478-99bd-a05fb1c396c8"). InnerVolumeSpecName "kube-api-access-jtv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:21:44.453641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.453596    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca" pod "4339d562-2402-4478-99bd-a05fb1c396c8" (UID: "4339d562-2402-4478-99bd-a05fb1c396c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:21:44.512646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.511211    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtv6v\" (UniqueName: \"kubernetes.io/projected/4339d562-2402-4478-99bd-a05fb1c396c8-kube-api-access-jtv6v\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:44.512646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.511255    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:44.512646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:44.511271    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4339d562-2402-4478-99bd-a05fb1c396c8-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:45.186889 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:45.186861    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"
Nov 01 07:21:45.187172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:45.187148    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp" event={"ID":"4339d562-2402-4478-99bd-a05fb1c396c8","Type":"ContainerDied","Data":"08e6324f45f448cbbd0b488115227ff7de01c784cf3b7ab8eee9d7d9d8925d2b"}
Nov 01 07:21:45.187357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:45.187340    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08e6324f45f448cbbd0b488115227ff7de01c784cf3b7ab8eee9d7d9d8925d2b"
Nov 01 07:21:45.827626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:45.827586    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:21:45.828285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:45.828249    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:21:51.381849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.381813    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"]
Nov 01 07:21:51.382328 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.382275    4468 cpu_manager.go:414] "containerMap: removing container" podUID="4339d562-2402-4478-99bd-a05fb1c396c8" containerName="revoke-keys"
Nov 01 07:21:51.382328 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.382286    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4339d562-2402-4478-99bd-a05fb1c396c8" containerName="revoke-keys"
Nov 01 07:21:51.382421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.382367    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="4339d562-2402-4478-99bd-a05fb1c396c8" containerName="revoke-keys"
Nov 01 07:21:51.385766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.385747    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.435274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.435233    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvzf\" (UniqueName: \"kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.435274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.435279    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.435549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.435397    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.442585 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.442554    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"]
Nov 01 07:21:51.488349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.488309    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:21:51.488560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.488357    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:21:51.488560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.488369    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:21:51.488560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.488362    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:21:51.536078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.536042    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvzf\" (UniqueName: \"kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.536288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.536093    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.536288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.536195    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.536928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.536908    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.538872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.538853    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.592570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.592536    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvzf\" (UniqueName: \"kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf\") pod \"maas-api-revoke-sub-2e58e3e4-rqxnh\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:51.698202 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:51.698119    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:52.068803 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:52.068775    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"]
Nov 01 07:21:52.070509 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:21:52.070459    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae52ff81_b7b7_47f8_8486_a65e5bc33382.slice/crio-147b19f5dbf5da1eb816d12a7cb86dd283948e32822b21377e0ad45d080c0587 WatchSource:0}: Error finding container 147b19f5dbf5da1eb816d12a7cb86dd283948e32822b21377e0ad45d080c0587: Status 404 returned error can't find the container with id 147b19f5dbf5da1eb816d12a7cb86dd283948e32822b21377e0ad45d080c0587
Nov 01 07:21:52.228536 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:52.228508    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh" event={"ID":"ae52ff81-b7b7-47f8-8486-a65e5bc33382","Type":"ContainerStarted","Data":"147b19f5dbf5da1eb816d12a7cb86dd283948e32822b21377e0ad45d080c0587"}
Nov 01 07:21:53.235363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:53.235267    4468 generic.go:356] "Generic (PLEG): container finished" podID="ae52ff81-b7b7-47f8-8486-a65e5bc33382" containerID="fc33477930f7b8880997b8aa426eae244b646a91c2a7ceaad58d8a7f78251d93" exitCode=0
Nov 01 07:21:53.235363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:53.235311    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh" event={"ID":"ae52ff81-b7b7-47f8-8486-a65e5bc33382","Type":"ContainerDied","Data":"fc33477930f7b8880997b8aa426eae244b646a91c2a7ceaad58d8a7f78251d93"}
Nov 01 07:21:54.391987 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.391952    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:54.464075 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.464041    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca\") pod \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") "
Nov 01 07:21:54.464326 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.464123    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf\" (UniqueName: \"kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf\") pod \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") "
Nov 01 07:21:54.464326 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.464241    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca\") pod \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\" (UID: \"ae52ff81-b7b7-47f8-8486-a65e5bc33382\") "
Nov 01 07:21:54.464550 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.464514    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca" pod "ae52ff81-b7b7-47f8-8486-a65e5bc33382" (UID: "ae52ff81-b7b7-47f8-8486-a65e5bc33382"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:21:54.464709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.464682    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:54.467179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.467146    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf" pod "ae52ff81-b7b7-47f8-8486-a65e5bc33382" (UID: "ae52ff81-b7b7-47f8-8486-a65e5bc33382"). InnerVolumeSpecName "kube-api-access-2gvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:21:54.467386 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.467363    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca" pod "ae52ff81-b7b7-47f8-8486-a65e5bc33382" (UID: "ae52ff81-b7b7-47f8-8486-a65e5bc33382"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:21:54.565760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.565713    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvzf\" (UniqueName: \"kubernetes.io/projected/ae52ff81-b7b7-47f8-8486-a65e5bc33382-kube-api-access-2gvzf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:54.565760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:54.565755    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae52ff81-b7b7-47f8-8486-a65e5bc33382-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:21:55.279807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:55.279697    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh" event={"ID":"ae52ff81-b7b7-47f8-8486-a65e5bc33382","Type":"ContainerDied","Data":"147b19f5dbf5da1eb816d12a7cb86dd283948e32822b21377e0ad45d080c0587"}
Nov 01 07:21:55.279807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:55.279760    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147b19f5dbf5da1eb816d12a7cb86dd283948e32822b21377e0ad45d080c0587"
Nov 01 07:21:55.279807 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:55.279713    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"
Nov 01 07:21:59.044142 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:59.044106    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"]
Nov 01 07:21:59.093561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:21:59.093527    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxlsp"]
Nov 01 07:22:00.588742 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:00.588712    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ea60f1-02b2-4556-8b1f-883ecd3957c7" path="/var/lib/kubelet/pods/25ea60f1-02b2-4556-8b1f-883ecd3957c7/volumes"
Nov 01 07:22:02.185738 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.185703    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"]
Nov 01 07:22:02.186285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.186172    4468 cpu_manager.go:414] "containerMap: removing container" podUID="ae52ff81-b7b7-47f8-8486-a65e5bc33382" containerName="revoke-keys"
Nov 01 07:22:02.186285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.186186    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae52ff81-b7b7-47f8-8486-a65e5bc33382" containerName="revoke-keys"
Nov 01 07:22:02.186285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.186268    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae52ff81-b7b7-47f8-8486-a65e5bc33382" containerName="revoke-keys"
Nov 01 07:22:02.189694 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.189678    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.250752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.250721    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5j4x\" (UniqueName: \"kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.250926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.250774    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.250926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.250874    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.292744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.292718    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:22:02.292744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.292741    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:22:02.292959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.292725    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:22:02.292959 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.292832    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:22:02.295247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.295227    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"]
Nov 01 07:22:02.352549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.352517    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.352720 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.352626    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5j4x\" (UniqueName: \"kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.352720 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.352672    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.353326 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.353297    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.355126 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.355109    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.409053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.409018    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5j4x\" (UniqueName: \"kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x\") pod \"maas-api-revoke-sub-83f582a1-vwp69\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.503454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.503372    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:02.878311 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:02.878282    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"]
Nov 01 07:22:02.880324 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:22:02.880294    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dde2fe7_b4de_4fc2_96f4_9146e863b580.slice/crio-3bab4172b278a11450a994c4d31c4c48cb58bdfc0df2c9386175e6249ee84bc1 WatchSource:0}: Error finding container 3bab4172b278a11450a994c4d31c4c48cb58bdfc0df2c9386175e6249ee84bc1: Status 404 returned error can't find the container with id 3bab4172b278a11450a994c4d31c4c48cb58bdfc0df2c9386175e6249ee84bc1
Nov 01 07:22:02.947106 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:22:02.947070    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:22:03.327893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:03.327854    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69" event={"ID":"0dde2fe7-b4de-4fc2-96f4-9146e863b580","Type":"ContainerStarted","Data":"9f1e141f369f4dce65fdc2dc346848781952e68e6f52ff1a994108ca8659745d"}
Nov 01 07:22:03.328343 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:03.327908    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69" event={"ID":"0dde2fe7-b4de-4fc2-96f4-9146e863b580","Type":"ContainerStarted","Data":"3bab4172b278a11450a994c4d31c4c48cb58bdfc0df2c9386175e6249ee84bc1"}
Nov 01 07:22:03.439973 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:03.439914    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69" podStartSLOduration=1.439895836 podStartE2EDuration="1.439895836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:22:03.439223166 +0000 UTC m=+4335.007029997" watchObservedRunningTime="2026-11-01 07:22:03.439895836 +0000 UTC m=+4335.007702666"
Nov 01 07:22:04.335510 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:04.335450    4468 generic.go:356] "Generic (PLEG): container finished" podID="0dde2fe7-b4de-4fc2-96f4-9146e863b580" containerID="9f1e141f369f4dce65fdc2dc346848781952e68e6f52ff1a994108ca8659745d" exitCode=0
Nov 01 07:22:04.335921 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:04.335536    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69" event={"ID":"0dde2fe7-b4de-4fc2-96f4-9146e863b580","Type":"ContainerDied","Data":"9f1e141f369f4dce65fdc2dc346848781952e68e6f52ff1a994108ca8659745d"}
Nov 01 07:22:05.518349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.518307    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:05.585813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.585777    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x\" (UniqueName: \"kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x\") pod \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") "
Nov 01 07:22:05.586040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.585843    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca\") pod \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") "
Nov 01 07:22:05.586040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.585939    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca\") pod \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\" (UID: \"0dde2fe7-b4de-4fc2-96f4-9146e863b580\") "
Nov 01 07:22:05.586624 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.586593    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca" pod "0dde2fe7-b4de-4fc2-96f4-9146e863b580" (UID: "0dde2fe7-b4de-4fc2-96f4-9146e863b580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:22:05.588729 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.588694    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x" pod "0dde2fe7-b4de-4fc2-96f4-9146e863b580" (UID: "0dde2fe7-b4de-4fc2-96f4-9146e863b580"). InnerVolumeSpecName "kube-api-access-s5j4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:05.588985 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.588964    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca" pod "0dde2fe7-b4de-4fc2-96f4-9146e863b580" (UID: "0dde2fe7-b4de-4fc2-96f4-9146e863b580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:05.688013 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.687860    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5j4x\" (UniqueName: \"kubernetes.io/projected/0dde2fe7-b4de-4fc2-96f4-9146e863b580-kube-api-access-s5j4x\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:05.688013 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.687894    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:05.688013 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:05.687910    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dde2fe7-b4de-4fc2-96f4-9146e863b580-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:06.385502 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:06.385118    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69" event={"ID":"0dde2fe7-b4de-4fc2-96f4-9146e863b580","Type":"ContainerDied","Data":"3bab4172b278a11450a994c4d31c4c48cb58bdfc0df2c9386175e6249ee84bc1"}
Nov 01 07:22:06.385502 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:06.385220    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bab4172b278a11450a994c4d31c4c48cb58bdfc0df2c9386175e6249ee84bc1"
Nov 01 07:22:06.385502 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:06.385320    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"
Nov 01 07:22:12.935441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:12.935405    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"]
Nov 01 07:22:12.935883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:12.935858    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0dde2fe7-b4de-4fc2-96f4-9146e863b580" containerName="revoke-keys"
Nov 01 07:22:12.935883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:12.935870    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dde2fe7-b4de-4fc2-96f4-9146e863b580" containerName="revoke-keys"
Nov 01 07:22:12.935967 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:12.935941    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dde2fe7-b4de-4fc2-96f4-9146e863b580" containerName="revoke-keys"
Nov 01 07:22:12.939371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:12.939354    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.041754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.041723    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:22:13.041971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.041782    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:22:13.041971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.041810    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:22:13.042250 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.042228    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:22:13.044902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.044876    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"]
Nov 01 07:22:13.087447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.087407    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74cq\" (UniqueName: \"kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.087681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.087638    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.087759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.087677    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.189491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.189377    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.189491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.189414    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.189706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.189497    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74cq\" (UniqueName: \"kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.190097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.190077    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.191989 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.191956    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.251962 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.247151    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74cq\" (UniqueName: \"kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq\") pod \"maas-api-revoke-sub-49729989-dwww5\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.252130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.252073    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:13.624394 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:13.624365    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"]
Nov 01 07:22:13.627297 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:22:13.627266    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37618ee_6ba7_4214_992a_96dfc3203822.slice/crio-faac4176cee957a5dd0c3924659ee6693134afe8c83b1a8c260ff6547dbb8835 WatchSource:0}: Error finding container faac4176cee957a5dd0c3924659ee6693134afe8c83b1a8c260ff6547dbb8835: Status 404 returned error can't find the container with id faac4176cee957a5dd0c3924659ee6693134afe8c83b1a8c260ff6547dbb8835
Nov 01 07:22:14.429322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:14.429287    4468 generic.go:356] "Generic (PLEG): container finished" podID="c37618ee-6ba7-4214-992a-96dfc3203822" containerID="77c200ead84ba97be65d806aff8e69a993b0a87fc0253391dda01663984a8d9e" exitCode=0
Nov 01 07:22:14.429716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:14.429331    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5" event={"ID":"c37618ee-6ba7-4214-992a-96dfc3203822","Type":"ContainerDied","Data":"77c200ead84ba97be65d806aff8e69a993b0a87fc0253391dda01663984a8d9e"}
Nov 01 07:22:14.429716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:14.429380    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5" event={"ID":"c37618ee-6ba7-4214-992a-96dfc3203822","Type":"ContainerStarted","Data":"faac4176cee957a5dd0c3924659ee6693134afe8c83b1a8c260ff6547dbb8835"}
Nov 01 07:22:15.573419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.573389    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:15.719530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.719422    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca\") pod \"c37618ee-6ba7-4214-992a-96dfc3203822\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") "
Nov 01 07:22:15.719719 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.719557    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq\" (UniqueName: \"kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq\") pod \"c37618ee-6ba7-4214-992a-96dfc3203822\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") "
Nov 01 07:22:15.719719 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.719637    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca\") pod \"c37618ee-6ba7-4214-992a-96dfc3203822\" (UID: \"c37618ee-6ba7-4214-992a-96dfc3203822\") "
Nov 01 07:22:15.720056 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.720025    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca" pod "c37618ee-6ba7-4214-992a-96dfc3203822" (UID: "c37618ee-6ba7-4214-992a-96dfc3203822"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:22:15.721768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.721745    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca" pod "c37618ee-6ba7-4214-992a-96dfc3203822" (UID: "c37618ee-6ba7-4214-992a-96dfc3203822"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:15.721875 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.721769    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq" pod "c37618ee-6ba7-4214-992a-96dfc3203822" (UID: "c37618ee-6ba7-4214-992a-96dfc3203822"). InnerVolumeSpecName "kube-api-access-x74cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:15.821164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.821129    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:15.821164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.821158    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37618ee-6ba7-4214-992a-96dfc3203822-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:15.821164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:15.821168    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74cq\" (UniqueName: \"kubernetes.io/projected/c37618ee-6ba7-4214-992a-96dfc3203822-kube-api-access-x74cq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:16.443093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:16.443046    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5" event={"ID":"c37618ee-6ba7-4214-992a-96dfc3203822","Type":"ContainerDied","Data":"faac4176cee957a5dd0c3924659ee6693134afe8c83b1a8c260ff6547dbb8835"}
Nov 01 07:22:16.443093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:16.443074    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"
Nov 01 07:22:16.443316 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:16.443104    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faac4176cee957a5dd0c3924659ee6693134afe8c83b1a8c260ff6547dbb8835"
Nov 01 07:22:18.334493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:18.333623    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:22:18.334493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:18.333749    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:22:18.439759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:18.439711    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:22:18.440002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:18.439797    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:22:23.683398 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.683363    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"]
Nov 01 07:22:23.683877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.683861    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c37618ee-6ba7-4214-992a-96dfc3203822" containerName="revoke-keys"
Nov 01 07:22:23.683925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.683879    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37618ee-6ba7-4214-992a-96dfc3203822" containerName="revoke-keys"
Nov 01 07:22:23.683973 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.683962    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37618ee-6ba7-4214-992a-96dfc3203822" containerName="revoke-keys"
Nov 01 07:22:23.687515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.687491    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.732596 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.732557    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.732780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.732710    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcgv\" (UniqueName: \"kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.732860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.732796    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.741950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.741924    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:22:23.744646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.744610    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"]
Nov 01 07:22:23.791370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.791338    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:22:23.791370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.791340    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:22:23.791370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.791376    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:22:23.834235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.834194    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcgv\" (UniqueName: \"kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.834432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.834254    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.834432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.834302    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.834968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.834949    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.836917 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.836891    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:23.890852 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:23.890817    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcgv\" (UniqueName: \"kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv\") pod \"maas-api-revoke-sub-367d4389-jsdb7\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:24.000820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:24.000736    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:24.375523 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:24.373216    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"]
Nov 01 07:22:24.507645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:24.507605    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7" event={"ID":"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61","Type":"ContainerStarted","Data":"80ccdc7b083c52e48ecdfb81eb62d1a29cfc4bd2b949da684ce9fba0fd361f3e"}
Nov 01 07:22:25.514007 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:25.513971    4468 generic.go:356] "Generic (PLEG): container finished" podID="f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" containerID="6a0e6f02915d68f43636d204679dc267e0eef1efd57814813add0d73807561cb" exitCode=0
Nov 01 07:22:25.514371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:25.514059    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7" event={"ID":"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61","Type":"ContainerDied","Data":"6a0e6f02915d68f43636d204679dc267e0eef1efd57814813add0d73807561cb"}
Nov 01 07:22:26.698529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.698495    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:26.766098 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.764889    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca\") pod \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") "
Nov 01 07:22:26.766098 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.764959    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv\" (UniqueName: \"kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv\") pod \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") "
Nov 01 07:22:26.766098 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.765008    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca\") pod \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\" (UID: \"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61\") "
Nov 01 07:22:26.766098 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.765783    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca" pod "f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" (UID: "f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:22:26.783599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.776583    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca" pod "f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" (UID: "f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:26.783599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.779985    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv" pod "f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" (UID: "f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61"). InnerVolumeSpecName "kube-api-access-lxcgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:26.870303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.870218    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:26.870303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.870258    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxcgv\" (UniqueName: \"kubernetes.io/projected/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-kube-api-access-lxcgv\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:26.870303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:26.870274    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:27.585606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:27.585568    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"
Nov 01 07:22:27.585606 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:27.585583    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7" event={"ID":"f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61","Type":"ContainerDied","Data":"80ccdc7b083c52e48ecdfb81eb62d1a29cfc4bd2b949da684ce9fba0fd361f3e"}
Nov 01 07:22:27.585825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:27.585630    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ccdc7b083c52e48ecdfb81eb62d1a29cfc4bd2b949da684ce9fba0fd361f3e"
Nov 01 07:22:32.948278 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:22:32.948189    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:22:36.788824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:36.788788    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:22:36.992702 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:36.992667    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"]
Nov 01 07:22:36.993139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:36.993123    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" containerName="revoke-keys"
Nov 01 07:22:36.993249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:36.993139    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" containerName="revoke-keys"
Nov 01 07:22:36.993249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:36.993246    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" containerName="revoke-keys"
Nov 01 07:22:36.996698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:36.996681    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.050001 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.049971    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:22:37.083072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.083036    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.083274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.083103    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6x7\" (UniqueName: \"kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.083274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.083194    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.099503 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.099449    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:22:37.099503 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.099499    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:22:37.099698 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.099521    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:22:37.102232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.102206    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"]
Nov 01 07:22:37.184620 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.184579    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.184818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.184641    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6x7\" (UniqueName: \"kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.184818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.184677    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.185300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.185276    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.187312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.187288    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.241129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.241091    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6x7\" (UniqueName: \"kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7\") pod \"maas-api-revoke-sub-def01b3b-fzmp6\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.310139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.310050    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:37.875324 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:37.875186    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"]
Nov 01 07:22:38.689130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:38.689088    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6" event={"ID":"1f771fb7-215e-4ef1-819d-10a5add3af99","Type":"ContainerStarted","Data":"55bd25bc0f2b2d57073b5dd54daa97f169e6ff1a7ef060301fa746aea1f7bd3e"}
Nov 01 07:22:38.689330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:38.689157    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6" event={"ID":"1f771fb7-215e-4ef1-819d-10a5add3af99","Type":"ContainerStarted","Data":"da6ae5287dac0a2e78e154913f444349bcea7f0d6ea0bc829987c597ddc09be1"}
Nov 01 07:22:38.806493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:38.804259    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6" podStartSLOduration=2.804237561 podStartE2EDuration="2.804237561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:22:38.800884777 +0000 UTC m=+4370.368691609" watchObservedRunningTime="2026-11-01 07:22:38.804237561 +0000 UTC m=+4370.372044392"
Nov 01 07:22:39.696338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:39.696294    4468 generic.go:356] "Generic (PLEG): container finished" podID="1f771fb7-215e-4ef1-819d-10a5add3af99" containerID="55bd25bc0f2b2d57073b5dd54daa97f169e6ff1a7ef060301fa746aea1f7bd3e" exitCode=0
Nov 01 07:22:39.696806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:39.696373    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6" event={"ID":"1f771fb7-215e-4ef1-819d-10a5add3af99","Type":"ContainerDied","Data":"55bd25bc0f2b2d57073b5dd54daa97f169e6ff1a7ef060301fa746aea1f7bd3e"}
Nov 01 07:22:40.843166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.843141    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:40.951615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.951577    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca\") pod \"1f771fb7-215e-4ef1-819d-10a5add3af99\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") "
Nov 01 07:22:40.951796 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.951714    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca\") pod \"1f771fb7-215e-4ef1-819d-10a5add3af99\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") "
Nov 01 07:22:40.951796 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.951744    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7\" (UniqueName: \"kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7\") pod \"1f771fb7-215e-4ef1-819d-10a5add3af99\" (UID: \"1f771fb7-215e-4ef1-819d-10a5add3af99\") "
Nov 01 07:22:40.951976 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.951953    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca" pod "1f771fb7-215e-4ef1-819d-10a5add3af99" (UID: "1f771fb7-215e-4ef1-819d-10a5add3af99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:22:40.952091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.952072    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:40.954253 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.954203    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca" pod "1f771fb7-215e-4ef1-819d-10a5add3af99" (UID: "1f771fb7-215e-4ef1-819d-10a5add3af99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:40.954441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:40.954415    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7" pod "1f771fb7-215e-4ef1-819d-10a5add3af99" (UID: "1f771fb7-215e-4ef1-819d-10a5add3af99"). InnerVolumeSpecName "kube-api-access-cw6x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:41.053526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:41.053491    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f771fb7-215e-4ef1-819d-10a5add3af99-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:41.053526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:41.053520    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6x7\" (UniqueName: \"kubernetes.io/projected/1f771fb7-215e-4ef1-819d-10a5add3af99-kube-api-access-cw6x7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:41.368952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:41.368911    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"]
Nov 01 07:22:41.712133 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:41.712048    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"
Nov 01 07:22:41.712133 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:41.712079    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6" event={"ID":"1f771fb7-215e-4ef1-819d-10a5add3af99","Type":"ContainerDied","Data":"da6ae5287dac0a2e78e154913f444349bcea7f0d6ea0bc829987c597ddc09be1"}
Nov 01 07:22:41.712133 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:41.712129    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6ae5287dac0a2e78e154913f444349bcea7f0d6ea0bc829987c597ddc09be1"
Nov 01 07:22:47.469914 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:47.469876    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:22:48.725211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.725178    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"]
Nov 01 07:22:48.725662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.725648    4468 cpu_manager.go:414] "containerMap: removing container" podUID="1f771fb7-215e-4ef1-819d-10a5add3af99" containerName="revoke-keys"
Nov 01 07:22:48.725713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.725663    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f771fb7-215e-4ef1-819d-10a5add3af99" containerName="revoke-keys"
Nov 01 07:22:48.725758 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.725748    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f771fb7-215e-4ef1-819d-10a5add3af99" containerName="revoke-keys"
Nov 01 07:22:48.729086 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.729071    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.831715 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.831674    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:22:48.831715 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.831696    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:22:48.831984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.831843    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\""
Nov 01 07:22:48.831984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.831934    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:22:48.834719 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.834699    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"]
Nov 01 07:22:48.842361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.839010    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.842361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.839053    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.842361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.839189    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672v5\" (UniqueName: \"kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.940788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.940750    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.940964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.940853    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672v5\" (UniqueName: \"kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.940964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.940921    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.941489 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.941440    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.943433 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.943401    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:48.997865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:48.997788    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-672v5\" (UniqueName: \"kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5\") pod \"maas-api-revoke-sub-c1f07235-jv87b\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:49.041565 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:49.041525    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:49.416884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:49.416858    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"]
Nov 01 07:22:49.418010 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:22:49.417986    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47934a10_f7a1_4ef7_973a_309013e936c4.slice/crio-4f84fafb0ff6129408201a8d980d8c6f7510b8aee0610b74790ad97ae94e71f5 WatchSource:0}: Error finding container 4f84fafb0ff6129408201a8d980d8c6f7510b8aee0610b74790ad97ae94e71f5: Status 404 returned error can't find the container with id 4f84fafb0ff6129408201a8d980d8c6f7510b8aee0610b74790ad97ae94e71f5
Nov 01 07:22:49.757768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:49.757722    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b" event={"ID":"47934a10-f7a1-4ef7-973a-309013e936c4","Type":"ContainerStarted","Data":"f40119f876a6d78072f72a2da51a66c86c0c64d749071ba4a4e3e608f7e091cc"}
Nov 01 07:22:49.758175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:49.757786    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b" event={"ID":"47934a10-f7a1-4ef7-973a-309013e936c4","Type":"ContainerStarted","Data":"4f84fafb0ff6129408201a8d980d8c6f7510b8aee0610b74790ad97ae94e71f5"}
Nov 01 07:22:49.873772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:49.873716    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b" podStartSLOduration=1.873698514 podStartE2EDuration="1.873698514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:22:49.871413006 +0000 UTC m=+4381.439219837" watchObservedRunningTime="2026-11-01 07:22:49.873698514 +0000 UTC m=+4381.441505343"
Nov 01 07:22:50.766162 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:50.766069    4468 generic.go:356] "Generic (PLEG): container finished" podID="47934a10-f7a1-4ef7-973a-309013e936c4" containerID="f40119f876a6d78072f72a2da51a66c86c0c64d749071ba4a4e3e608f7e091cc" exitCode=0
Nov 01 07:22:50.766588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:50.766149    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b" event={"ID":"47934a10-f7a1-4ef7-973a-309013e936c4","Type":"ContainerDied","Data":"f40119f876a6d78072f72a2da51a66c86c0c64d749071ba4a4e3e608f7e091cc"}
Nov 01 07:22:51.909752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:51.909718    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:52.077658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.077623    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca\") pod \"47934a10-f7a1-4ef7-973a-309013e936c4\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") "
Nov 01 07:22:52.077844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.077801    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5\" (UniqueName: \"kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5\") pod \"47934a10-f7a1-4ef7-973a-309013e936c4\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") "
Nov 01 07:22:52.077844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.077826    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca\") pod \"47934a10-f7a1-4ef7-973a-309013e936c4\" (UID: \"47934a10-f7a1-4ef7-973a-309013e936c4\") "
Nov 01 07:22:52.078006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.077985    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca" pod "47934a10-f7a1-4ef7-973a-309013e936c4" (UID: "47934a10-f7a1-4ef7-973a-309013e936c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:22:52.078141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.078126    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:52.080190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.080164    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca" pod "47934a10-f7a1-4ef7-973a-309013e936c4" (UID: "47934a10-f7a1-4ef7-973a-309013e936c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:52.080321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.080165    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5" pod "47934a10-f7a1-4ef7-973a-309013e936c4" (UID: "47934a10-f7a1-4ef7-973a-309013e936c4"). InnerVolumeSpecName "kube-api-access-672v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:52.178963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.178924    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672v5\" (UniqueName: \"kubernetes.io/projected/47934a10-f7a1-4ef7-973a-309013e936c4-kube-api-access-672v5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:52.178963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.178956    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47934a10-f7a1-4ef7-973a-309013e936c4-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:52.779025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.778985    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b" event={"ID":"47934a10-f7a1-4ef7-973a-309013e936c4","Type":"ContainerDied","Data":"4f84fafb0ff6129408201a8d980d8c6f7510b8aee0610b74790ad97ae94e71f5"}
Nov 01 07:22:52.779212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.779035    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f84fafb0ff6129408201a8d980d8c6f7510b8aee0610b74790ad97ae94e71f5"
Nov 01 07:22:52.779212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:52.779003    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"
Nov 01 07:22:56.389623 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.389556    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr" podUID="a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" containerName="main" containerID="cri-o://a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef" gracePeriod=45
Nov 01 07:22:56.666409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.666378    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:22:56.804372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.804338    4468 generic.go:356] "Generic (PLEG): container finished" podID="a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" containerID="a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef" exitCode=0
Nov 01 07:22:56.804601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.804388    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"
Nov 01 07:22:56.804601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.804417    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr" event={"ID":"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f","Type":"ContainerDied","Data":"a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef"}
Nov 01 07:22:56.804601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.804479    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr" event={"ID":"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f","Type":"ContainerDied","Data":"a1ba448428fadf76c1c88515578f11f845a720c3f58d055c7a5e28dd827b32a2"}
Nov 01 07:22:56.804601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.804492    4468 scope.go:122] "RemoveContainer" containerID="a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef"
Nov 01 07:22:56.816100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.816069    4468 scope.go:122] "RemoveContainer" containerID="a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef"
Nov 01 07:22:56.816544 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:22:56.816515    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef\": container with ID starting with a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef not found: ID does not exist" containerID="a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef"
Nov 01 07:22:56.816636 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.816561    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef"} err="failed to get container status \"a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef\": rpc error: code = NotFound desc = could not find container \"a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef\": container with ID starting with a76cf2e86bd32f5d589a8d216e9aa5d3c758a52327441dfb9ff794342a1df8ef not found: ID does not exist"
Nov 01 07:22:56.827858 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.827829    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm\") pod \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") "
Nov 01 07:22:56.828002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.827877    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache\") pod \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") "
Nov 01 07:22:56.828002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.827923    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir\") pod \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") "
Nov 01 07:22:56.828002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.827947    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home\") pod \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") "
Nov 01 07:22:56.828002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.827975    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs\") pod \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") "
Nov 01 07:22:56.828245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.828103    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw\" (UniqueName: \"kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw\") pod \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\" (UID: \"a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f\") "
Nov 01 07:22:56.828245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.828217    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home" pod "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" (UID: "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:22:56.828245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.828231    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir" pod "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" (UID: "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:22:56.828526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.828497    4468 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tmp-dir\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:56.828526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.828527    4468 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-home\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:56.828756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.828607    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache" pod "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" (UID: "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:22:56.830769 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.830740    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm" pod "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" (UID: "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:22:56.830891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.830832    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs" pod "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" (UID: "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:22:56.833599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.833573    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw" pod "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" (UID: "a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f"). InnerVolumeSpecName "kube-api-access-dd6sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:22:56.929703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.929618    4468 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-dshm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:56.929703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.929650    4468 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-model-cache\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:56.929703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.929660    4468 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-tls-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:56.929703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:56.929671    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6sw\" (UniqueName: \"kubernetes.io/projected/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f-kube-api-access-dd6sw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:22:57.223987 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:57.223891    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"]
Nov 01 07:22:57.226494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:57.226454    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-main-976689/premium-main-976689-kserve-69ddf894f-q8ddr"]
Nov 01 07:22:58.589383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:22:58.589347    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" path="/var/lib/kubelet/pods/a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f/volumes"
Nov 01 07:23:02.948073 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:23:02.948030    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:23:03.888611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:03.888564    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:23:14.773845 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.773806    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"]
Nov 01 07:23:14.774315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.774239    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" containerName="main"
Nov 01 07:23:14.774315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.774250    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" containerName="main"
Nov 01 07:23:14.774315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.774260    4468 cpu_manager.go:414] "containerMap: removing container" podUID="47934a10-f7a1-4ef7-973a-309013e936c4" containerName="revoke-keys"
Nov 01 07:23:14.774315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.774265    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47934a10-f7a1-4ef7-973a-309013e936c4" containerName="revoke-keys"
Nov 01 07:23:14.774456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.774339    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1307d6e-a515-4e2b-b1cb-7b462cbb1c2f" containerName="main"
Nov 01 07:23:14.774456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.774348    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="47934a10-f7a1-4ef7-973a-309013e936c4" containerName="revoke-keys"
Nov 01 07:23:14.777957 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.777934    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.830417 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.830380    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-976689-openshift-default-dockercfg-k9llf\""
Nov 01 07:23:14.884284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.884254    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"]
Nov 01 07:23:14.911560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911528    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911565    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911789 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911593    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911789 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911741    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911789    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911837    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6p78\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911927 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911885    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911966 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911933    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:14.911966 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:14.911958    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012562    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012613    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012636    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012678    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012714    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012742    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6p78\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012776    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012810    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.012887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.012834    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.013266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.013025    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.013266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.013150    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.013266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.013244    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.013406 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.013285    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.013595 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.013579    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.015298 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.015266    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.015496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.015450    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.069400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.069372    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.069586 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.069527    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6p78\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-nkwj5\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.090404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.090374    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:15.163415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.163377    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:23:15.515136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.515106    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"]
Nov 01 07:23:15.515891 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:23:15.515866    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdbc7a8_d1aa_4fe3_a0a3_1ff2e783fe83.slice/crio-95f153af58c6a5ba80bbd5ba7ad29a338d2298414f73d1e5375f6380a1d00e52 WatchSource:0}: Error finding container 95f153af58c6a5ba80bbd5ba7ad29a338d2298414f73d1e5375f6380a1d00e52: Status 404 returned error can't find the container with id 95f153af58c6a5ba80bbd5ba7ad29a338d2298414f73d1e5375f6380a1d00e52
Nov 01 07:23:15.518032 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.517999    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:23:15.518108 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.518067    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:23:15.518108 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.518097    4468 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 07:23:15.912027 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.911988    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" event={"ID":"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83","Type":"ContainerStarted","Data":"d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef"}
Nov 01 07:23:15.912459 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:15.912057    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" event={"ID":"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83","Type":"ContainerStarted","Data":"95f153af58c6a5ba80bbd5ba7ad29a338d2298414f73d1e5375f6380a1d00e52"}
Nov 01 07:23:16.090745 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:16.090700    4468 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:16.092214 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:16.092190    4468 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 07:23:16.092329 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:16.092248    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 07:23:17.091028 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:17.090994    4468 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 07:23:17.091458 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:17.091043    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 07:23:18.091685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:18.091647    4468 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 07:23:18.092102 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:18.091706    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 07:23:19.091607 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:19.091566    4468 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 07:23:19.091808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:19.091621    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 07:23:20.090946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:20.090905    4468 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 07:23:20.091155 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:20.090967    4468 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 07:23:21.029128 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:21.029077    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podStartSLOduration=7.029062367 podStartE2EDuration="7.029062367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:16.035788878 +0000 UTC m=+4407.603595713" watchObservedRunningTime="2026-11-01 07:23:21.029062367 +0000 UTC m=+4412.596869197"
Nov 01 07:23:21.030381 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:21.030351    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"]
Nov 01 07:23:21.030592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:21.030566    4468 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy" containerID="cri-o://d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef" gracePeriod=30
Nov 01 07:23:26.895983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:26.895957    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:26.996768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:26.996671    4468 generic.go:356] "Generic (PLEG): container finished" podID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerID="d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef" exitCode=0
Nov 01 07:23:26.996768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:26.996725    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"
Nov 01 07:23:26.996971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:26.996762    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" event={"ID":"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83","Type":"ContainerDied","Data":"d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef"}
Nov 01 07:23:26.996971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:26.996814    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5" event={"ID":"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83","Type":"ContainerDied","Data":"95f153af58c6a5ba80bbd5ba7ad29a338d2298414f73d1e5375f6380a1d00e52"}
Nov 01 07:23:26.996971 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:26.996836    4468 scope.go:122] "RemoveContainer" containerID="d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef"
Nov 01 07:23:27.006198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.006181    4468 scope.go:122] "RemoveContainer" containerID="d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef"
Nov 01 07:23:27.006522 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:23:27.006502    4468 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef\": container with ID starting with d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef not found: ID does not exist" containerID="d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef"
Nov 01 07:23:27.006593 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.006534    4468 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef"} err="failed to get container status \"d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef\": rpc error: code = NotFound desc = could not find container \"d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef\": container with ID starting with d8d7cc9ea1553e29e159bc5eff3f506af58f9953924292845167f59b8eb2daef not found: ID does not exist"
Nov 01 07:23:27.048165 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048124    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048219    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048247    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048305    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048355 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048353    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048390    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048417    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048449    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048519    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert\") pod \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\" (UID: \"9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83\") "
Nov 01 07:23:27.048611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048552    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:23:27.048887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048739    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:23:27.048887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048835    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:23:27.048887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.048845    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:23:27.049050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.049001    4468 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-credential-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.049050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.049024    4468 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-socket\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.049050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.049039    4468 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-workload-certs\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.049166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.049054    4468 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-data\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.049247 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.049219    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:23:27.050716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.050692    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:23:27.050716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.050697    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:23:27.051025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.051005    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:23:27.051112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.051069    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78" pod "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" (UID: "9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83"). InnerVolumeSpecName "kube-api-access-h6p78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:23:27.150653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.150616    4468 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-podinfo\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.150653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.150648    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6p78\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-kube-api-access-h6p78\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.150653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.150658    4468 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-token\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.150929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.150667    4468 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istiod-ca-cert\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.150929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.150676    4468 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83-istio-envoy\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:27.417183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.417142    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"]
Nov 01 07:23:27.420461 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:27.420431    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-nkwj5"]
Nov 01 07:23:28.589758 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:28.589721    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" path="/var/lib/kubelet/pods/9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83/volumes"
Nov 01 07:23:32.949548 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:23:32.949513    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:23:43.001834 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:43.001799    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:23:52.744183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:23:52.744143    4468 scope.go:122] "RemoveContainer" containerID="80b7b9736a26f3b7a95d13d8504a1bd603e047c83278b5b37bec1290dbab7969"
Nov 01 07:24:02.947037 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:24:02.946951    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:24:05.544037 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.543994    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"]
Nov 01 07:24:05.544456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.544447    4468 cpu_manager.go:414] "containerMap: removing container" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy"
Nov 01 07:24:05.544549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.544458    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy"
Nov 01 07:24:05.544549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.544546    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdbc7a8-d1aa-4fe3-a0a3-1ff2e783fe83" containerName="istio-proxy"
Nov 01 07:24:05.547723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.547706    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.605611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.605578    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"]
Nov 01 07:24:05.623589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.623559    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhvc\" (UniqueName: \"kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.623778 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.623745    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.623824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.623797    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.651359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.651316    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:24:05.651359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.651339    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:24:05.651359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.651326    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:24:05.651359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.651331    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:24:05.725115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.725070    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.725338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.725131    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.725338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.725193    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhvc\" (UniqueName: \"kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.725898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.725879    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.727679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.727659    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.781574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.781541    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhvc\" (UniqueName: \"kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc\") pod \"maas-api-revoke-keys-e2e-ait-501c44d6-s9gks\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:05.860854 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:05.860817    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:06.231270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:06.231244    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"]
Nov 01 07:24:06.233426 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:24:06.233398    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43cda99_bf07_4765_a01a_3f999e7f9f66.slice/crio-ba5790abb441fcdeaa1f9f14fb128003be3028fb51d2c6c0f86a3a654e7ade70 WatchSource:0}: Error finding container ba5790abb441fcdeaa1f9f14fb128003be3028fb51d2c6c0f86a3a654e7ade70: Status 404 returned error can't find the container with id ba5790abb441fcdeaa1f9f14fb128003be3028fb51d2c6c0f86a3a654e7ade70
Nov 01 07:24:07.245275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:07.245240    4468 generic.go:356] "Generic (PLEG): container finished" podID="c43cda99-bf07-4765-a01a-3f999e7f9f66" containerID="de71c28627954390bd960940b02313426d681185c51602d62e3e8a943dd692ae" exitCode=0
Nov 01 07:24:07.245733 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:07.245303    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks" event={"ID":"c43cda99-bf07-4765-a01a-3f999e7f9f66","Type":"ContainerDied","Data":"de71c28627954390bd960940b02313426d681185c51602d62e3e8a943dd692ae"}
Nov 01 07:24:07.245733 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:07.245343    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks" event={"ID":"c43cda99-bf07-4765-a01a-3f999e7f9f66","Type":"ContainerStarted","Data":"ba5790abb441fcdeaa1f9f14fb128003be3028fb51d2c6c0f86a3a654e7ade70"}
Nov 01 07:24:08.386321 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.386296    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:08.455745 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.455708    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc\" (UniqueName: \"kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc\") pod \"c43cda99-bf07-4765-a01a-3f999e7f9f66\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") "
Nov 01 07:24:08.455945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.455801    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca\") pod \"c43cda99-bf07-4765-a01a-3f999e7f9f66\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") "
Nov 01 07:24:08.455945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.455886    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca\") pod \"c43cda99-bf07-4765-a01a-3f999e7f9f66\" (UID: \"c43cda99-bf07-4765-a01a-3f999e7f9f66\") "
Nov 01 07:24:08.456148 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.456125    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca" pod "c43cda99-bf07-4765-a01a-3f999e7f9f66" (UID: "c43cda99-bf07-4765-a01a-3f999e7f9f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:08.456235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.456213    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:08.458019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.457995    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca" pod "c43cda99-bf07-4765-a01a-3f999e7f9f66" (UID: "c43cda99-bf07-4765-a01a-3f999e7f9f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:08.458129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.458067    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc" pod "c43cda99-bf07-4765-a01a-3f999e7f9f66" (UID: "c43cda99-bf07-4765-a01a-3f999e7f9f66"). InnerVolumeSpecName "kube-api-access-lnhvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:08.557713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.557681    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhvc\" (UniqueName: \"kubernetes.io/projected/c43cda99-bf07-4765-a01a-3f999e7f9f66-kube-api-access-lnhvc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:08.557713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:08.557710    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43cda99-bf07-4765-a01a-3f999e7f9f66-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:09.257760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:09.257732    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"
Nov 01 07:24:09.257942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:09.257756    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks" event={"ID":"c43cda99-bf07-4765-a01a-3f999e7f9f66","Type":"ContainerDied","Data":"ba5790abb441fcdeaa1f9f14fb128003be3028fb51d2c6c0f86a3a654e7ade70"}
Nov 01 07:24:09.257942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:09.257815    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba5790abb441fcdeaa1f9f14fb128003be3028fb51d2c6c0f86a3a654e7ade70"
Nov 01 07:24:19.045589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:19.045555    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"]
Nov 01 07:24:19.093877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:19.093838    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flrch"]
Nov 01 07:24:20.588996 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:20.588960    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf72aa70-148c-4b18-9387-b5e4f215a88a" path="/var/lib/kubelet/pods/bf72aa70-148c-4b18-9387-b5e4f215a88a/volumes"
Nov 01 07:24:24.787574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:24.787532    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"]
Nov 01 07:24:24.837046 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:24.837012    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-501c44d6-s9gks"]
Nov 01 07:24:26.589555 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:26.589520    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43cda99-bf07-4765-a01a-3f999e7f9f66" path="/var/lib/kubelet/pods/c43cda99-bf07-4765-a01a-3f999e7f9f66/volumes"
Nov 01 07:24:27.265231 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:27.265189    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:24:32.948540 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:24:32.948505    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:24:37.306925 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:37.306887    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:24:38.825432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.825393    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"]
Nov 01 07:24:38.826045 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.826023    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c43cda99-bf07-4765-a01a-3f999e7f9f66" containerName="revoke-keys"
Nov 01 07:24:38.826139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.826048    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43cda99-bf07-4765-a01a-3f999e7f9f66" containerName="revoke-keys"
Nov 01 07:24:38.826197 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.826164    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43cda99-bf07-4765-a01a-3f999e7f9f66" containerName="revoke-keys"
Nov 01 07:24:38.829414 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.829393    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:38.932126 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.932091    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:24:38.932126 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.932111    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:24:38.932378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.932149    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:24:38.932378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.932185    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:24:38.935426 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:38.935402    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"]
Nov 01 07:24:39.005784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.005742    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpjq\" (UniqueName: \"kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.005784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.005789    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.006078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.005969    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.107345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.107235    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.107345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.107324    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpjq\" (UniqueName: \"kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.107641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.107361    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.107957 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.107933    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.109970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.109948    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.164142 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.164111    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpjq\" (UniqueName: \"kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq\") pod \"maas-api-revoke-sub-f904bff9-r5bbb\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.442280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.442197    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:39.812863 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:39.812827    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"]
Nov 01 07:24:39.814593 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:24:39.814563    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf770f40e_e825_4643_acf7_fdc79a38dafc.slice/crio-a4b145a6d109a3d85217601aa4a272e4b72e89b79bb4b69a20f0a57377c4fabd WatchSource:0}: Error finding container a4b145a6d109a3d85217601aa4a272e4b72e89b79bb4b69a20f0a57377c4fabd: Status 404 returned error can't find the container with id a4b145a6d109a3d85217601aa4a272e4b72e89b79bb4b69a20f0a57377c4fabd
Nov 01 07:24:40.495312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:40.495203    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb" event={"ID":"f770f40e-e825-4643-acf7-fdc79a38dafc","Type":"ContainerStarted","Data":"9253bb1b9a523bb644769797812d7bcdf6f512b6dac2f4ef5fe2256260205a50"}
Nov 01 07:24:40.495312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:40.495266    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb" event={"ID":"f770f40e-e825-4643-acf7-fdc79a38dafc","Type":"ContainerStarted","Data":"a4b145a6d109a3d85217601aa4a272e4b72e89b79bb4b69a20f0a57377c4fabd"}
Nov 01 07:24:40.607393 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:40.607268    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb" podStartSLOduration=2.607248663 podStartE2EDuration="2.607248663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:24:40.605967294 +0000 UTC m=+4492.173774125" watchObservedRunningTime="2026-11-01 07:24:40.607248663 +0000 UTC m=+4492.175055494"
Nov 01 07:24:41.501995 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:41.501951    4468 generic.go:356] "Generic (PLEG): container finished" podID="f770f40e-e825-4643-acf7-fdc79a38dafc" containerID="9253bb1b9a523bb644769797812d7bcdf6f512b6dac2f4ef5fe2256260205a50" exitCode=0
Nov 01 07:24:41.502439 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:41.501998    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb" event={"ID":"f770f40e-e825-4643-acf7-fdc79a38dafc","Type":"ContainerDied","Data":"9253bb1b9a523bb644769797812d7bcdf6f512b6dac2f4ef5fe2256260205a50"}
Nov 01 07:24:42.658151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.658117    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:42.847217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.847180    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq\" (UniqueName: \"kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq\") pod \"f770f40e-e825-4643-acf7-fdc79a38dafc\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") "
Nov 01 07:24:42.847408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.847259    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca\") pod \"f770f40e-e825-4643-acf7-fdc79a38dafc\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") "
Nov 01 07:24:42.847408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.847303    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca\") pod \"f770f40e-e825-4643-acf7-fdc79a38dafc\" (UID: \"f770f40e-e825-4643-acf7-fdc79a38dafc\") "
Nov 01 07:24:42.847653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.847631    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca" pod "f770f40e-e825-4643-acf7-fdc79a38dafc" (UID: "f770f40e-e825-4643-acf7-fdc79a38dafc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:42.849699 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.849671    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq" pod "f770f40e-e825-4643-acf7-fdc79a38dafc" (UID: "f770f40e-e825-4643-acf7-fdc79a38dafc"). InnerVolumeSpecName "kube-api-access-mlpjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:42.849821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.849720    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca" pod "f770f40e-e825-4643-acf7-fdc79a38dafc" (UID: "f770f40e-e825-4643-acf7-fdc79a38dafc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:42.948310 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.948272    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:42.948310 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.948305    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlpjq\" (UniqueName: \"kubernetes.io/projected/f770f40e-e825-4643-acf7-fdc79a38dafc-kube-api-access-mlpjq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:42.948310 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:42.948316    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f770f40e-e825-4643-acf7-fdc79a38dafc-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:43.516103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:43.516070    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"
Nov 01 07:24:43.516103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:43.516096    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb" event={"ID":"f770f40e-e825-4643-acf7-fdc79a38dafc","Type":"ContainerDied","Data":"a4b145a6d109a3d85217601aa4a272e4b72e89b79bb4b69a20f0a57377c4fabd"}
Nov 01 07:24:43.516334 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:43.516141    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b145a6d109a3d85217601aa4a272e4b72e89b79bb4b69a20f0a57377c4fabd"
Nov 01 07:24:51.674941 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:24:51.674902    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:25:00.980072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:00.980028    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:25:02.947800 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:25:02.947762    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:25:09.975301 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:09.975261    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:25:11.622918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.622884    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"]
Nov 01 07:25:11.623364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.623334    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f770f40e-e825-4643-acf7-fdc79a38dafc" containerName="revoke-keys"
Nov 01 07:25:11.623364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.623347    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f770f40e-e825-4643-acf7-fdc79a38dafc" containerName="revoke-keys"
Nov 01 07:25:11.623448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.623424    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f770f40e-e825-4643-acf7-fdc79a38dafc" containerName="revoke-keys"
Nov 01 07:25:11.626957 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.626932    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.684276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.684242    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"]
Nov 01 07:25:11.719656 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.719615    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.719656 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.719651    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.719881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.719676    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb66g\" (UniqueName: \"kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.730239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.730208    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:25:11.730239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.730230    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:25:11.730239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.730232    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:25:11.730481 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.730294    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:25:11.820952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.820909    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.820952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.820951    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.821183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.820974    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb66g\" (UniqueName: \"kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.821599 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.821576    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.823559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.823530    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.877332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.877254    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb66g\" (UniqueName: \"kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g\") pod \"maas-api-revoke-sub-f03629db-djw8s\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:11.941008 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:11.940966    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:12.317456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:12.317417    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"]
Nov 01 07:25:12.326185 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:25:12.326156    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2140c7ac_a64e_4a9e_893d_1af0b992f2b8.slice/crio-f5dbd8e40d89929f3d38ad19c49f6941655607cfbaf573801f09bb1fcc56a989 WatchSource:0}: Error finding container f5dbd8e40d89929f3d38ad19c49f6941655607cfbaf573801f09bb1fcc56a989: Status 404 returned error can't find the container with id f5dbd8e40d89929f3d38ad19c49f6941655607cfbaf573801f09bb1fcc56a989
Nov 01 07:25:12.722156 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:12.722117    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s" event={"ID":"2140c7ac-a64e-4a9e-893d-1af0b992f2b8","Type":"ContainerStarted","Data":"29bcb14cbf07e291d397107aa977a503f8d1c9e4bd28037ba85df57c1256727e"}
Nov 01 07:25:12.722718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:12.722183    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s" event={"ID":"2140c7ac-a64e-4a9e-893d-1af0b992f2b8","Type":"ContainerStarted","Data":"f5dbd8e40d89929f3d38ad19c49f6941655607cfbaf573801f09bb1fcc56a989"}
Nov 01 07:25:12.837040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:12.836982    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s" podStartSLOduration=1.836965829 podStartE2EDuration="1.836965829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:25:12.835060634 +0000 UTC m=+4524.402867476" watchObservedRunningTime="2026-11-01 07:25:12.836965829 +0000 UTC m=+4524.404772658"
Nov 01 07:25:13.728612 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:13.728577    4468 generic.go:356] "Generic (PLEG): container finished" podID="2140c7ac-a64e-4a9e-893d-1af0b992f2b8" containerID="29bcb14cbf07e291d397107aa977a503f8d1c9e4bd28037ba85df57c1256727e" exitCode=0
Nov 01 07:25:13.729024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:13.728628    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s" event={"ID":"2140c7ac-a64e-4a9e-893d-1af0b992f2b8","Type":"ContainerDied","Data":"29bcb14cbf07e291d397107aa977a503f8d1c9e4bd28037ba85df57c1256727e"}
Nov 01 07:25:14.873109 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.873079    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:14.952298 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.952259    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca\") pod \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") "
Nov 01 07:25:14.952500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.952375    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g\" (UniqueName: \"kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g\") pod \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") "
Nov 01 07:25:14.952500 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.952418    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca\") pod \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\" (UID: \"2140c7ac-a64e-4a9e-893d-1af0b992f2b8\") "
Nov 01 07:25:14.952659 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.952635    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca" pod "2140c7ac-a64e-4a9e-893d-1af0b992f2b8" (UID: "2140c7ac-a64e-4a9e-893d-1af0b992f2b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:25:14.952816 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.952794    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:14.954666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.954645    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca" pod "2140c7ac-a64e-4a9e-893d-1af0b992f2b8" (UID: "2140c7ac-a64e-4a9e-893d-1af0b992f2b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:25:14.954758 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:14.954706    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g" pod "2140c7ac-a64e-4a9e-893d-1af0b992f2b8" (UID: "2140c7ac-a64e-4a9e-893d-1af0b992f2b8"). InnerVolumeSpecName "kube-api-access-hb66g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:25:15.054118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:15.054078    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:15.054118 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:15.054110    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb66g\" (UniqueName: \"kubernetes.io/projected/2140c7ac-a64e-4a9e-893d-1af0b992f2b8-kube-api-access-hb66g\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:15.741537 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:15.741496    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s" event={"ID":"2140c7ac-a64e-4a9e-893d-1af0b992f2b8","Type":"ContainerDied","Data":"f5dbd8e40d89929f3d38ad19c49f6941655607cfbaf573801f09bb1fcc56a989"}
Nov 01 07:25:15.741537 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:15.741528    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"
Nov 01 07:25:15.741751 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:15.741556    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5dbd8e40d89929f3d38ad19c49f6941655607cfbaf573801f09bb1fcc56a989"
Nov 01 07:25:17.044230 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:17.044194    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"]
Nov 01 07:25:17.093777 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:17.093742    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e85756a3-mvnhm"]
Nov 01 07:25:18.210342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:18.210301    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:25:18.589067 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:18.589033    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c761959-c16a-4ba6-9d14-b66f5ec10ca3" path="/var/lib/kubelet/pods/7c761959-c16a-4ba6-9d14-b66f5ec10ca3/volumes"
Nov 01 07:25:28.050919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:28.050863    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"]
Nov 01 07:25:28.100922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:28.100879    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe4a15f-ds9z6"]
Nov 01 07:25:28.589739 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:28.589695    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35f0354-6517-462d-9248-6df4dcaa4e76" path="/var/lib/kubelet/pods/c35f0354-6517-462d-9248-6df4dcaa4e76/volumes"
Nov 01 07:25:30.800910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.800832    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"]
Nov 01 07:25:30.801316 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.801246    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2140c7ac-a64e-4a9e-893d-1af0b992f2b8" containerName="revoke-keys"
Nov 01 07:25:30.801316 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.801257    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2140c7ac-a64e-4a9e-893d-1af0b992f2b8" containerName="revoke-keys"
Nov 01 07:25:30.801404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.801344    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2140c7ac-a64e-4a9e-893d-1af0b992f2b8" containerName="revoke-keys"
Nov 01 07:25:30.804605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.804585    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:30.861689 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.861657    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"]
Nov 01 07:25:30.907713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.907679    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:25:30.907713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.907698    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:25:30.907949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.907717    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:25:30.907949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.907757    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:25:30.950591 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.950547    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:30.950765 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.950613    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsj4j\" (UniqueName: \"kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:30.950765 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:30.950748    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.052436 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.052336    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.052673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.052519    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.052673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.052574    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsj4j\" (UniqueName: \"kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.053164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.053133    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.055166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.055144    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.109412 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.109382    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsj4j\" (UniqueName: \"kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j\") pod \"maas-api-revoke-sub-0165d7d2-mx2h5\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.117638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.117611    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:31.493611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.493582    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"]
Nov 01 07:25:31.495042 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:25:31.495002    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08532d8_767d_4417_a47a_22ae56cb2c1c.slice/crio-f4c80aea0383261180cd0dde893b0bce3a74aa4e271d40bda6dd0ae326c67e28 WatchSource:0}: Error finding container f4c80aea0383261180cd0dde893b0bce3a74aa4e271d40bda6dd0ae326c67e28: Status 404 returned error can't find the container with id f4c80aea0383261180cd0dde893b0bce3a74aa4e271d40bda6dd0ae326c67e28
Nov 01 07:25:31.856661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.856628    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5" event={"ID":"c08532d8-767d-4417-a47a-22ae56cb2c1c","Type":"ContainerStarted","Data":"765d7197ba4c35d78b417d096447f9178073673b79a4b3b57fe29d38831195e0"}
Nov 01 07:25:31.857047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.856687    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5" event={"ID":"c08532d8-767d-4417-a47a-22ae56cb2c1c","Type":"ContainerStarted","Data":"f4c80aea0383261180cd0dde893b0bce3a74aa4e271d40bda6dd0ae326c67e28"}
Nov 01 07:25:31.969424 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:31.969367    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5" podStartSLOduration=1.96935215 podStartE2EDuration="1.96935215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:25:31.967885857 +0000 UTC m=+4543.535692687" watchObservedRunningTime="2026-11-01 07:25:31.96935215 +0000 UTC m=+4543.537158979"
Nov 01 07:25:32.863127 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:32.863093    4468 generic.go:356] "Generic (PLEG): container finished" podID="c08532d8-767d-4417-a47a-22ae56cb2c1c" containerID="765d7197ba4c35d78b417d096447f9178073673b79a4b3b57fe29d38831195e0" exitCode=0
Nov 01 07:25:32.863644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:32.863180    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5" event={"ID":"c08532d8-767d-4417-a47a-22ae56cb2c1c","Type":"ContainerDied","Data":"765d7197ba4c35d78b417d096447f9178073673b79a4b3b57fe29d38831195e0"}
Nov 01 07:25:32.947856 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:25:32.947823    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:25:34.013901 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.013875    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:34.084226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.084187    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j\" (UniqueName: \"kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j\") pod \"c08532d8-767d-4417-a47a-22ae56cb2c1c\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") "
Nov 01 07:25:34.084408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.084324    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca\") pod \"c08532d8-767d-4417-a47a-22ae56cb2c1c\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") "
Nov 01 07:25:34.084487 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.084446    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca\") pod \"c08532d8-767d-4417-a47a-22ae56cb2c1c\" (UID: \"c08532d8-767d-4417-a47a-22ae56cb2c1c\") "
Nov 01 07:25:34.084691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.084667    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca" pod "c08532d8-767d-4417-a47a-22ae56cb2c1c" (UID: "c08532d8-767d-4417-a47a-22ae56cb2c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:25:34.084809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.084795    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:34.086605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.086580    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca" pod "c08532d8-767d-4417-a47a-22ae56cb2c1c" (UID: "c08532d8-767d-4417-a47a-22ae56cb2c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:25:34.086723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.086624    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j" pod "c08532d8-767d-4417-a47a-22ae56cb2c1c" (UID: "c08532d8-767d-4417-a47a-22ae56cb2c1c"). InnerVolumeSpecName "kube-api-access-xsj4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:25:34.185362 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.185285    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08532d8-767d-4417-a47a-22ae56cb2c1c-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:34.185362 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.185311    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsj4j\" (UniqueName: \"kubernetes.io/projected/c08532d8-767d-4417-a47a-22ae56cb2c1c-kube-api-access-xsj4j\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:34.874998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.874916    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"
Nov 01 07:25:34.874998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.874940    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5" event={"ID":"c08532d8-767d-4417-a47a-22ae56cb2c1c","Type":"ContainerDied","Data":"f4c80aea0383261180cd0dde893b0bce3a74aa4e271d40bda6dd0ae326c67e28"}
Nov 01 07:25:34.874998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:34.874982    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c80aea0383261180cd0dde893b0bce3a74aa4e271d40bda6dd0ae326c67e28"
Nov 01 07:25:42.490634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.490595    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"]
Nov 01 07:25:42.491068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.491020    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c08532d8-767d-4417-a47a-22ae56cb2c1c" containerName="revoke-keys"
Nov 01 07:25:42.491068 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.491031    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08532d8-767d-4417-a47a-22ae56cb2c1c" containerName="revoke-keys"
Nov 01 07:25:42.491144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.491106    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08532d8-767d-4417-a47a-22ae56cb2c1c" containerName="revoke-keys"
Nov 01 07:25:42.494377 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.494357    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.551372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.551338    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"]
Nov 01 07:25:42.597517 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.597482    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:25:42.597517 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.597490    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:25:42.597717 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.597489    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:25:42.597717 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.597540    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:25:42.675632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.675597    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.675814 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.675663    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.675814 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.675718    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.777277 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.777175    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.777616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.777367    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.777616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.777409    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.778117 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.778091    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.779901 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.779878    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:42.834408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:42.834364    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth\") pod \"maas-api-revoke-sub-6b19338f-p262b\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:43.107014 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:43.106966    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:43.477808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:43.477781    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"]
Nov 01 07:25:43.479415 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:25:43.479387    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb20bb44_9136_4e79_b62b_c040ff65fbfb.slice/crio-e67678bbf710a71ccd5911d51a7ecb9b9d50d649a3b1c805e2189d05be217b32 WatchSource:0}: Error finding container e67678bbf710a71ccd5911d51a7ecb9b9d50d649a3b1c805e2189d05be217b32: Status 404 returned error can't find the container with id e67678bbf710a71ccd5911d51a7ecb9b9d50d649a3b1c805e2189d05be217b32
Nov 01 07:25:43.928136 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:43.928099    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b" event={"ID":"db20bb44-9136-4e79-b62b-c040ff65fbfb","Type":"ContainerStarted","Data":"1616d040de2d63d3af7f2d2b2589a44332111000b3fc98a3ca9957b586ea9797"}
Nov 01 07:25:43.928679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:43.928174    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b" event={"ID":"db20bb44-9136-4e79-b62b-c040ff65fbfb","Type":"ContainerStarted","Data":"e67678bbf710a71ccd5911d51a7ecb9b9d50d649a3b1c805e2189d05be217b32"}
Nov 01 07:25:44.042114 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:44.042056    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b" podStartSLOduration=2.042041578 podStartE2EDuration="2.042041578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:25:44.039657803 +0000 UTC m=+4555.607464636" watchObservedRunningTime="2026-11-01 07:25:44.042041578 +0000 UTC m=+4555.609848408"
Nov 01 07:25:44.935117 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:44.935078    4468 generic.go:356] "Generic (PLEG): container finished" podID="db20bb44-9136-4e79-b62b-c040ff65fbfb" containerID="1616d040de2d63d3af7f2d2b2589a44332111000b3fc98a3ca9957b586ea9797" exitCode=0
Nov 01 07:25:44.935704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:44.935168    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b" event={"ID":"db20bb44-9136-4e79-b62b-c040ff65fbfb","Type":"ContainerDied","Data":"1616d040de2d63d3af7f2d2b2589a44332111000b3fc98a3ca9957b586ea9797"}
Nov 01 07:25:46.081309 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.081282    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:46.211327 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.211241    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca\") pod \"db20bb44-9136-4e79-b62b-c040ff65fbfb\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") "
Nov 01 07:25:46.211547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.211344    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca\") pod \"db20bb44-9136-4e79-b62b-c040ff65fbfb\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") "
Nov 01 07:25:46.211547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.211499    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth\") pod \"db20bb44-9136-4e79-b62b-c040ff65fbfb\" (UID: \"db20bb44-9136-4e79-b62b-c040ff65fbfb\") "
Nov 01 07:25:46.211791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.211764    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca" pod "db20bb44-9136-4e79-b62b-c040ff65fbfb" (UID: "db20bb44-9136-4e79-b62b-c040ff65fbfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:25:46.211931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.211913    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:46.213645 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.213620    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca" pod "db20bb44-9136-4e79-b62b-c040ff65fbfb" (UID: "db20bb44-9136-4e79-b62b-c040ff65fbfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:25:46.213769 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.213750    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth" pod "db20bb44-9136-4e79-b62b-c040ff65fbfb" (UID: "db20bb44-9136-4e79-b62b-c040ff65fbfb"). InnerVolumeSpecName "kube-api-access-bwrth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:25:46.312786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.312753    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db20bb44-9136-4e79-b62b-c040ff65fbfb-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:46.312786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.312780    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/db20bb44-9136-4e79-b62b-c040ff65fbfb-kube-api-access-bwrth\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:25:46.948643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.948560    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b" event={"ID":"db20bb44-9136-4e79-b62b-c040ff65fbfb","Type":"ContainerDied","Data":"e67678bbf710a71ccd5911d51a7ecb9b9d50d649a3b1c805e2189d05be217b32"}
Nov 01 07:25:46.948643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.948612    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67678bbf710a71ccd5911d51a7ecb9b9d50d649a3b1c805e2189d05be217b32"
Nov 01 07:25:46.948643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:46.948568    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"
Nov 01 07:25:49.050427 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:49.050391    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"]
Nov 01 07:25:49.094425 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:49.094387    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbabb089-rqzzw"]
Nov 01 07:25:50.590376 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:50.590342    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9" path="/var/lib/kubelet/pods/f22b6a2e-a553-4ae6-bbb5-4cff8d3ed9d9/volumes"
Nov 01 07:25:52.783836 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:52.783807    4468 scope.go:122] "RemoveContainer" containerID="f2a3cb9e1c106a476ed20d28866004ba9464560d24a409afc8fb64c8c1d9a145"
Nov 01 07:25:55.307419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:25:55.300960    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:26:01.050813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:01.050770    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"]
Nov 01 07:26:01.101281 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:01.101202    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1346d1b-lnj6l"]
Nov 01 07:26:02.589267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:02.589226    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be6894e-4348-4199-a3dd-c59b83bee976" path="/var/lib/kubelet/pods/6be6894e-4348-4199-a3dd-c59b83bee976/volumes"
Nov 01 07:26:02.991727 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:26:02.991627    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:26:03.329646 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:03.329596    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:26:03.329997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:03.329963    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:26:07.176249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:07.176211    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:26:12.045495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:12.045420    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"]
Nov 01 07:26:12.094908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:12.094874    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb27aa15-zvlh5"]
Nov 01 07:26:12.589395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:12.589352    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a64407-8623-46ed-9e57-efa8509bb9e0" path="/var/lib/kubelet/pods/77a64407-8623-46ed-9e57-efa8509bb9e0/volumes"
Nov 01 07:26:23.046141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:23.046107    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"]
Nov 01 07:26:23.095487 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:23.095425    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff87d5e-kccjc"]
Nov 01 07:26:24.588701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:24.588662    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7e0379-bd90-4161-a50d-7f8558adb485" path="/var/lib/kubelet/pods/1b7e0379-bd90-4161-a50d-7f8558adb485/volumes"
Nov 01 07:26:32.947774 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:26:32.947737    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:26:34.045964 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:34.045931    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"]
Nov 01 07:26:34.094898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:34.094862    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f6480b-6gl9n"]
Nov 01 07:26:34.589334 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:34.589300    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dda460f-8a3d-43a6-a907-523d9262adbd" path="/var/lib/kubelet/pods/2dda460f-8a3d-43a6-a907-523d9262adbd/volumes"
Nov 01 07:26:45.090132 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:45.090086    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"]
Nov 01 07:26:45.092513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:45.092486    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00e18343-lrfhp"]
Nov 01 07:26:46.361648 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:46.361614    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:26:46.588725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:46.588693    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4339d562-2402-4478-99bd-a05fb1c396c8" path="/var/lib/kubelet/pods/4339d562-2402-4478-99bd-a05fb1c396c8/volumes"
Nov 01 07:26:52.807012 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:52.806973    4468 scope.go:122] "RemoveContainer" containerID="7592283ec4f11fcb4de4967b7244acf9bb72fb2a6c5ea204c58612737c201652"
Nov 01 07:26:52.816059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:52.816031    4468 scope.go:122] "RemoveContainer" containerID="5c064786a3d0077d50f54d670aaf143ea1dcda4761df63d1715daa21421f0445"
Nov 01 07:26:52.824665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:52.824645    4468 scope.go:122] "RemoveContainer" containerID="191fb4dafbeb4bf59af0ee6230dd387a7ebc01f0f4db95f77503558550a59657"
Nov 01 07:26:55.043937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:55.043903    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"]
Nov 01 07:26:55.093525 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:55.093481    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e58e3e4-rqxnh"]
Nov 01 07:26:56.589632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:56.589592    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae52ff81-b7b7-47f8-8486-a65e5bc33382" path="/var/lib/kubelet/pods/ae52ff81-b7b7-47f8-8486-a65e5bc33382/volumes"
Nov 01 07:26:57.348083 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.348047    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"]
Nov 01 07:26:57.348569 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.348555    4468 cpu_manager.go:414] "containerMap: removing container" podUID="db20bb44-9136-4e79-b62b-c040ff65fbfb" containerName="revoke-keys"
Nov 01 07:26:57.348621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.348570    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db20bb44-9136-4e79-b62b-c040ff65fbfb" containerName="revoke-keys"
Nov 01 07:26:57.348668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.348640    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="db20bb44-9136-4e79-b62b-c040ff65fbfb" containerName="revoke-keys"
Nov 01 07:26:57.353236 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.353214    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.408211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.408178    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"]
Nov 01 07:26:57.448909 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.448865    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zdg\" (UniqueName: \"kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.448909 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.448906    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.449151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.448996    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.454937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.454910    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:26:57.454937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.454911    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:26:57.455095 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.454953    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:26:57.455095 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.455003    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:26:57.550288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.550242    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zdg\" (UniqueName: \"kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.550288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.550283    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.550567 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.550326    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.551018 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.550996    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.552994 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.552970    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.606933 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.606856    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zdg\" (UniqueName: \"kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg\") pod \"maas-api-revoke-sub-e011430c-hckpq\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:57.666673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:57.666632    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:26:58.039259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:58.039228    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"]
Nov 01 07:26:58.040557 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:26:58.040528    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7362266b_1eaf_484e_b273_088f89d76f05.slice/crio-4423be7e5d37021346a0a559e5993361bf7e061f85aeca7b42837fc1c1b5f817 WatchSource:0}: Error finding container 4423be7e5d37021346a0a559e5993361bf7e061f85aeca7b42837fc1c1b5f817: Status 404 returned error can't find the container with id 4423be7e5d37021346a0a559e5993361bf7e061f85aeca7b42837fc1c1b5f817
Nov 01 07:26:58.408209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:58.408167    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq" event={"ID":"7362266b-1eaf-484e-b273-088f89d76f05","Type":"ContainerStarted","Data":"a6b0daf23347e0346ae24bdf690e70bfeac664b58c2c1de84d9e6403251a2fee"}
Nov 01 07:26:58.408398 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:58.408224    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq" event={"ID":"7362266b-1eaf-484e-b273-088f89d76f05","Type":"ContainerStarted","Data":"4423be7e5d37021346a0a559e5993361bf7e061f85aeca7b42837fc1c1b5f817"}
Nov 01 07:26:58.523069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:58.523020    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq" podStartSLOduration=1.5230039560000002 podStartE2EDuration="1.523003956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:26:58.521893202 +0000 UTC m=+4630.089700033" watchObservedRunningTime="2026-11-01 07:26:58.523003956 +0000 UTC m=+4630.090810786"
Nov 01 07:26:59.415130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:59.415091    4468 generic.go:356] "Generic (PLEG): container finished" podID="7362266b-1eaf-484e-b273-088f89d76f05" containerID="a6b0daf23347e0346ae24bdf690e70bfeac664b58c2c1de84d9e6403251a2fee" exitCode=0
Nov 01 07:26:59.415719 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:26:59.415184    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq" event={"ID":"7362266b-1eaf-484e-b273-088f89d76f05","Type":"ContainerDied","Data":"a6b0daf23347e0346ae24bdf690e70bfeac664b58c2c1de84d9e6403251a2fee"}
Nov 01 07:27:00.586554 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.586513    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:27:00.683305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.683204    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg\" (UniqueName: \"kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg\") pod \"7362266b-1eaf-484e-b273-088f89d76f05\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") "
Nov 01 07:27:00.683305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.683278    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca\") pod \"7362266b-1eaf-484e-b273-088f89d76f05\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") "
Nov 01 07:27:00.683555 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.683325    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca\") pod \"7362266b-1eaf-484e-b273-088f89d76f05\" (UID: \"7362266b-1eaf-484e-b273-088f89d76f05\") "
Nov 01 07:27:00.683806 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.683776    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca" pod "7362266b-1eaf-484e-b273-088f89d76f05" (UID: "7362266b-1eaf-484e-b273-088f89d76f05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:27:00.685664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.685638    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg" pod "7362266b-1eaf-484e-b273-088f89d76f05" (UID: "7362266b-1eaf-484e-b273-088f89d76f05"). InnerVolumeSpecName "kube-api-access-n6zdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:27:00.685758 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.685636    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca" pod "7362266b-1eaf-484e-b273-088f89d76f05" (UID: "7362266b-1eaf-484e-b273-088f89d76f05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:27:00.785049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.785003    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:27:00.785049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.785039    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7362266b-1eaf-484e-b273-088f89d76f05-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:27:00.785049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:00.785052    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zdg\" (UniqueName: \"kubernetes.io/projected/7362266b-1eaf-484e-b273-088f89d76f05-kube-api-access-n6zdg\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:27:01.434953 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:01.434924    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"
Nov 01 07:27:01.435140 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:01.434970    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq" event={"ID":"7362266b-1eaf-484e-b273-088f89d76f05","Type":"ContainerDied","Data":"4423be7e5d37021346a0a559e5993361bf7e061f85aeca7b42837fc1c1b5f817"}
Nov 01 07:27:01.435140 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:01.435035    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4423be7e5d37021346a0a559e5993361bf7e061f85aeca7b42837fc1c1b5f817"
Nov 01 07:27:02.947224 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:27:02.947187    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:27:06.043799 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:06.043765    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"]
Nov 01 07:27:06.100955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:06.096045    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f582a1-vwp69"]
Nov 01 07:27:06.588898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:06.588860    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dde2fe7-b4de-4fc2-96f4-9146e863b580" path="/var/lib/kubelet/pods/0dde2fe7-b4de-4fc2-96f4-9146e863b580/volumes"
Nov 01 07:27:10.381517 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:10.381456    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:27:15.816325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.816282    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"]
Nov 01 07:27:15.817010 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.816989    4468 cpu_manager.go:414] "containerMap: removing container" podUID="7362266b-1eaf-484e-b273-088f89d76f05" containerName="revoke-keys"
Nov 01 07:27:15.817061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.817015    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7362266b-1eaf-484e-b273-088f89d76f05" containerName="revoke-keys"
Nov 01 07:27:15.817157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.817144    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="7362266b-1eaf-484e-b273-088f89d76f05" containerName="revoke-keys"
Nov 01 07:27:15.821903 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.821876    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:15.879626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.879591    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"]
Nov 01 07:27:15.924445 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.924405    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:27:15.924730 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.924416    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:27:15.924892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.924444    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:27:15.924892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.924507    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:27:15.937931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.937902    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688ts\" (UniqueName: \"kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:15.938078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.937944    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:15.938078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:15.937978    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.039097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.039054    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688ts\" (UniqueName: \"kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.039097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.039094    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.039415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.039131    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.040004 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.039977    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.042144 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.042123    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.047679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.047648    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"]
Nov 01 07:27:16.051332 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.051304    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49729989-dwww5"]
Nov 01 07:27:16.096279 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.096202    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-688ts\" (UniqueName: \"kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts\") pod \"maas-api-revoke-sub-ef6e2041-hntbw\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.139561 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.139521    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:16.554949 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.554893    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"]
Nov 01 07:27:16.591127 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:16.591094    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37618ee-6ba7-4214-992a-96dfc3203822" path="/var/lib/kubelet/pods/c37618ee-6ba7-4214-992a-96dfc3203822/volumes"
Nov 01 07:27:17.572685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:17.572644    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw" event={"ID":"b6766fee-93a4-4559-8669-f2d47140ecc4","Type":"ContainerStarted","Data":"ebf6f8802efda8703c4dfde9778d20182d91ae37cd88544fe764ad690dff7d04"}
Nov 01 07:27:17.573343 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:17.573317    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw" event={"ID":"b6766fee-93a4-4559-8669-f2d47140ecc4","Type":"ContainerStarted","Data":"f460c9a0fd3fe4557e76dc1206807674d8374940844ff94bc48a3ff37339c3d8"}
Nov 01 07:27:17.696588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:17.696536    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw" podStartSLOduration=2.696519625 podStartE2EDuration="2.696519625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:27:17.693751052 +0000 UTC m=+4649.261557883" watchObservedRunningTime="2026-11-01 07:27:17.696519625 +0000 UTC m=+4649.264326454"
Nov 01 07:27:18.579372 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:18.579340    4468 generic.go:356] "Generic (PLEG): container finished" podID="b6766fee-93a4-4559-8669-f2d47140ecc4" containerID="ebf6f8802efda8703c4dfde9778d20182d91ae37cd88544fe764ad690dff7d04" exitCode=0
Nov 01 07:27:18.579853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:18.579433    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw" event={"ID":"b6766fee-93a4-4559-8669-f2d47140ecc4","Type":"ContainerDied","Data":"ebf6f8802efda8703c4dfde9778d20182d91ae37cd88544fe764ad690dff7d04"}
Nov 01 07:27:19.724208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.724180    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:19.809546 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.809504    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts\" (UniqueName: \"kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts\") pod \"b6766fee-93a4-4559-8669-f2d47140ecc4\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") "
Nov 01 07:27:19.809756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.809588    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca\") pod \"b6766fee-93a4-4559-8669-f2d47140ecc4\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") "
Nov 01 07:27:19.809756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.809632    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca\") pod \"b6766fee-93a4-4559-8669-f2d47140ecc4\" (UID: \"b6766fee-93a4-4559-8669-f2d47140ecc4\") "
Nov 01 07:27:19.809990 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.809966    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca" pod "b6766fee-93a4-4559-8669-f2d47140ecc4" (UID: "b6766fee-93a4-4559-8669-f2d47140ecc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:27:19.811842 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.811813    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca" pod "b6766fee-93a4-4559-8669-f2d47140ecc4" (UID: "b6766fee-93a4-4559-8669-f2d47140ecc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:27:19.811981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.811930    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts" pod "b6766fee-93a4-4559-8669-f2d47140ecc4" (UID: "b6766fee-93a4-4559-8669-f2d47140ecc4"). InnerVolumeSpecName "kube-api-access-688ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:27:19.910615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.910515    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-688ts\" (UniqueName: \"kubernetes.io/projected/b6766fee-93a4-4559-8669-f2d47140ecc4-kube-api-access-688ts\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:27:19.910615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.910558    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:27:19.910615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:19.910569    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6766fee-93a4-4559-8669-f2d47140ecc4-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:27:20.591291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:20.591250    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw" event={"ID":"b6766fee-93a4-4559-8669-f2d47140ecc4","Type":"ContainerDied","Data":"f460c9a0fd3fe4557e76dc1206807674d8374940844ff94bc48a3ff37339c3d8"}
Nov 01 07:27:20.591291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:20.591270    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"
Nov 01 07:27:20.591580 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:20.591304    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f460c9a0fd3fe4557e76dc1206807674d8374940844ff94bc48a3ff37339c3d8"
Nov 01 07:27:21.367396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:21.367355    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:27:27.044932 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:27.044900    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"]
Nov 01 07:27:27.094428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:27.094397    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367d4389-jsdb7"]
Nov 01 07:27:28.588681 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:28.588643    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61" path="/var/lib/kubelet/pods/f776fd6f-48b8-4dc4-82ce-42d0c5ff0a61/volumes"
Nov 01 07:27:32.948218 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:27:32.948178    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:27:40.270574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:40.270540    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:27:41.045504 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:41.045433    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"]
Nov 01 07:27:41.093819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:41.093783    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def01b3b-fzmp6"]
Nov 01 07:27:42.589254 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:42.589208    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f771fb7-215e-4ef1-819d-10a5add3af99" path="/var/lib/kubelet/pods/1f771fb7-215e-4ef1-819d-10a5add3af99/volumes"
Nov 01 07:27:52.045240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.045207    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"]
Nov 01 07:27:52.047180 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.047158    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f07235-jv87b"]
Nov 01 07:27:52.589174 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.589140    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47934a10-f7a1-4ef7-973a-309013e936c4" path="/var/lib/kubelet/pods/47934a10-f7a1-4ef7-973a-309013e936c4/volumes"
Nov 01 07:27:52.856344 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.856259    4468 scope.go:122] "RemoveContainer" containerID="fc33477930f7b8880997b8aa426eae244b646a91c2a7ceaad58d8a7f78251d93"
Nov 01 07:27:52.865423 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.865401    4468 scope.go:122] "RemoveContainer" containerID="72b37b634b893f939a513d58c429acdc12bfcdf6602db545701e3cf8eed6b8d5"
Nov 01 07:27:52.873682 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.873662    4468 scope.go:122] "RemoveContainer" containerID="46479d5e8b95013d78088ea58510cb42f304d7cf73aa8dfddaede2272b332e21"
Nov 01 07:27:52.881798 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.881778    4468 scope.go:122] "RemoveContainer" containerID="ed8cbaae077420f278db8e63f1196e66744c924aca8728022802c1cfdc37adca"
Nov 01 07:27:52.889544 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.889521    4468 scope.go:122] "RemoveContainer" containerID="0772cf96d1737472b266199c44b38a77222adf21aac5cec528f29a84eb2108d2"
Nov 01 07:27:52.897226 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:27:52.897210    4468 scope.go:122] "RemoveContainer" containerID="eeb9f1549151ec37ef7a85dd4c30850d85496b758c01ef5bbf9ea4457674af2c"
Nov 01 07:28:02.947859 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:28:02.947820    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:28:08.266895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:08.266860    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:28:32.947246 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:28:32.947172    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:28:38.392100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.392066    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"]
Nov 01 07:28:38.392588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.392506    4468 cpu_manager.go:414] "containerMap: removing container" podUID="b6766fee-93a4-4559-8669-f2d47140ecc4" containerName="revoke-keys"
Nov 01 07:28:38.392588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.392517    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6766fee-93a4-4559-8669-f2d47140ecc4" containerName="revoke-keys"
Nov 01 07:28:38.392676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.392594    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6766fee-93a4-4559-8669-f2d47140ecc4" containerName="revoke-keys"
Nov 01 07:28:38.395614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.395597    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.452190 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.452158    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"]
Nov 01 07:28:38.467699 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.467661    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.468110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.467725    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4g7l\" (UniqueName: \"kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.468110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.467860    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.498857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.498819    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:28:38.498857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.498819    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:28:38.498857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.498831    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:28:38.499113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.498878    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:28:38.569266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.569233    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.569513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.569338    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.569513 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.569380    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4g7l\" (UniqueName: \"kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.569997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.569975    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.572218 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.572193    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.625867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.625831    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4g7l\" (UniqueName: \"kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l\") pod \"maas-api-revoke-sub-452b6d7b-9b6zz\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:38.707926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:38.707841    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:39.078217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:39.078182    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"]
Nov 01 07:28:39.081231 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:28:39.081194    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a4ccc0_83c7_4bd4_b225_1d819c66f1a5.slice/crio-17a1a0fc8f4dd494c4f0c658659de2b30fc3ba895a269abb71673ea48d953595 WatchSource:0}: Error finding container 17a1a0fc8f4dd494c4f0c658659de2b30fc3ba895a269abb71673ea48d953595: Status 404 returned error can't find the container with id 17a1a0fc8f4dd494c4f0c658659de2b30fc3ba895a269abb71673ea48d953595
Nov 01 07:28:39.122735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:39.122705    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz" event={"ID":"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5","Type":"ContainerStarted","Data":"17a1a0fc8f4dd494c4f0c658659de2b30fc3ba895a269abb71673ea48d953595"}
Nov 01 07:28:39.700028 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:39.699944    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:28:40.129559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:40.129526    4468 generic.go:356] "Generic (PLEG): container finished" podID="00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" containerID="13eac24d722d86776fc9bef588f297e12738e856e88567d7bc708ffcecc4f46a" exitCode=0
Nov 01 07:28:40.129713 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:40.129613    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz" event={"ID":"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5","Type":"ContainerDied","Data":"13eac24d722d86776fc9bef588f297e12738e856e88567d7bc708ffcecc4f46a"}
Nov 01 07:28:41.399647 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.399574    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:41.523506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.516390    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca\") pod \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") "
Nov 01 07:28:41.523506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.516448    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca\") pod \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") "
Nov 01 07:28:41.523506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.516528    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l\" (UniqueName: \"kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l\") pod \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\" (UID: \"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5\") "
Nov 01 07:28:41.523506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.519900    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca" pod "00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" (UID: "00a4ccc0-83c7-4bd4-b225-1d819c66f1a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:28:41.537493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.534606    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l" pod "00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" (UID: "00a4ccc0-83c7-4bd4-b225-1d819c66f1a5"). InnerVolumeSpecName "kube-api-access-n4g7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:28:41.537493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.536886    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca" pod "00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" (UID: "00a4ccc0-83c7-4bd4-b225-1d819c66f1a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:28:41.626491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.620162    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:28:41.626491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.620210    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:28:41.626491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:41.620229    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4g7l\" (UniqueName: \"kubernetes.io/projected/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5-kube-api-access-n4g7l\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:28:42.185419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:42.182982    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz" event={"ID":"00a4ccc0-83c7-4bd4-b225-1d819c66f1a5","Type":"ContainerDied","Data":"17a1a0fc8f4dd494c4f0c658659de2b30fc3ba895a269abb71673ea48d953595"}
Nov 01 07:28:42.185419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:42.183045    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17a1a0fc8f4dd494c4f0c658659de2b30fc3ba895a269abb71673ea48d953595"
Nov 01 07:28:42.185419 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:42.183120    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"
Nov 01 07:28:52.936824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:52.936792    4468 scope.go:122] "RemoveContainer" containerID="f40119f876a6d78072f72a2da51a66c86c0c64d749071ba4a4e3e608f7e091cc"
Nov 01 07:28:52.945664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:52.945644    4468 scope.go:122] "RemoveContainer" containerID="55bd25bc0f2b2d57073b5dd54daa97f169e6ff1a7ef060301fa746aea1f7bd3e"
Nov 01 07:28:52.954110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:52.954088    4468 scope.go:122] "RemoveContainer" containerID="6a0e6f02915d68f43636d204679dc267e0eef1efd57814813add0d73807561cb"
Nov 01 07:28:52.962167 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:52.962148    4468 scope.go:122] "RemoveContainer" containerID="77c200ead84ba97be65d806aff8e69a993b0a87fc0253391dda01663984a8d9e"
Nov 01 07:28:52.970041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:52.970016    4468 scope.go:122] "RemoveContainer" containerID="9f1e141f369f4dce65fdc2dc346848781952e68e6f52ff1a994108ca8659745d"
Nov 01 07:28:55.694572 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:28:55.694526    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:29:02.947685 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:29:02.947648    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:29:07.720952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.720918    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"]
Nov 01 07:29:07.721614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.721594    4468 cpu_manager.go:414] "containerMap: removing container" podUID="00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" containerName="revoke-keys"
Nov 01 07:29:07.721723 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.721615    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" containerName="revoke-keys"
Nov 01 07:29:07.721790 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.721723    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" containerName="revoke-keys"
Nov 01 07:29:07.724931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.724910    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.761110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.761074    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.761292 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.761126    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdn9\" (UniqueName: \"kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.761292 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.761264    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.827672 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.827634    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:29:07.827672 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.827657    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:29:07.827672 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.827657    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:29:07.827953 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.827865    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:29:07.830005 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.829985    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"]
Nov 01 07:29:07.862493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.862430    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.862687 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.862550    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.862687 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.862581    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdn9\" (UniqueName: \"kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.863266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.863230    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.865115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.865095    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:07.918826 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:07.918792    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdn9\" (UniqueName: \"kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9\") pod \"maas-api-revoke-sub-3e31a506-2jqwt\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:08.040907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:08.040822    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:08.410568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:08.410540    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"]
Nov 01 07:29:08.412523 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:29:08.412496    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049949da_bcb1_44fd_beef_a88164bbbf8a.slice/crio-7b41a6425c122cc796f091dbe78b96287843116bedce92cc1232e730d997093d WatchSource:0}: Error finding container 7b41a6425c122cc796f091dbe78b96287843116bedce92cc1232e730d997093d: Status 404 returned error can't find the container with id 7b41a6425c122cc796f091dbe78b96287843116bedce92cc1232e730d997093d
Nov 01 07:29:08.473375 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:08.473344    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:29:09.341946 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:09.341907    4468 generic.go:356] "Generic (PLEG): container finished" podID="049949da-bcb1-44fd-beef-a88164bbbf8a" containerID="7dc75375193b020ad832689f32d8850deb56f062acd24eebabadeae486abce47" exitCode=0
Nov 01 07:29:09.342422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:09.342042    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt" event={"ID":"049949da-bcb1-44fd-beef-a88164bbbf8a","Type":"ContainerDied","Data":"7dc75375193b020ad832689f32d8850deb56f062acd24eebabadeae486abce47"}
Nov 01 07:29:09.342422 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:09.342112    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt" event={"ID":"049949da-bcb1-44fd-beef-a88164bbbf8a","Type":"ContainerStarted","Data":"7b41a6425c122cc796f091dbe78b96287843116bedce92cc1232e730d997093d"}
Nov 01 07:29:10.749940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.749649    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:10.795173 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.795121    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca\") pod \"049949da-bcb1-44fd-beef-a88164bbbf8a\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") "
Nov 01 07:29:10.795356 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.795188    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9\" (UniqueName: \"kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9\") pod \"049949da-bcb1-44fd-beef-a88164bbbf8a\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") "
Nov 01 07:29:10.795356 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.795300    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca\") pod \"049949da-bcb1-44fd-beef-a88164bbbf8a\" (UID: \"049949da-bcb1-44fd-beef-a88164bbbf8a\") "
Nov 01 07:29:10.816064 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.796113    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca" pod "049949da-bcb1-44fd-beef-a88164bbbf8a" (UID: "049949da-bcb1-44fd-beef-a88164bbbf8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:29:10.880082 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.879996    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9" pod "049949da-bcb1-44fd-beef-a88164bbbf8a" (UID: "049949da-bcb1-44fd-beef-a88164bbbf8a"). InnerVolumeSpecName "kube-api-access-tkdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:29:10.882605 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.882556    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca" pod "049949da-bcb1-44fd-beef-a88164bbbf8a" (UID: "049949da-bcb1-44fd-beef-a88164bbbf8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:29:10.897219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.897194    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:29:10.897219 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.897221    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049949da-bcb1-44fd-beef-a88164bbbf8a-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:29:10.897361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:10.897231    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdn9\" (UniqueName: \"kubernetes.io/projected/049949da-bcb1-44fd-beef-a88164bbbf8a-kube-api-access-tkdn9\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:29:11.361306 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:11.361276    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"
Nov 01 07:29:11.361548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:11.361275    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt" event={"ID":"049949da-bcb1-44fd-beef-a88164bbbf8a","Type":"ContainerDied","Data":"7b41a6425c122cc796f091dbe78b96287843116bedce92cc1232e730d997093d"}
Nov 01 07:29:11.361548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:11.361428    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b41a6425c122cc796f091dbe78b96287843116bedce92cc1232e730d997093d"
Nov 01 07:29:21.092290 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:21.092257    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:29:30.601113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:30.601079    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:29:32.618446 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.618416    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"]
Nov 01 07:29:32.618887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.618854    4468 cpu_manager.go:414] "containerMap: removing container" podUID="049949da-bcb1-44fd-beef-a88164bbbf8a" containerName="revoke-keys"
Nov 01 07:29:32.618887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.618867    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049949da-bcb1-44fd-beef-a88164bbbf8a" containerName="revoke-keys"
Nov 01 07:29:32.618988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.618953    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="049949da-bcb1-44fd-beef-a88164bbbf8a" containerName="revoke-keys"
Nov 01 07:29:32.622039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.622022    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.677692 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.677629    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:29:32.702228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.702194    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzgz\" (UniqueName: \"kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.702394 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.702266    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.702447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.702430    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.727228 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.727196    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:29:32.727358 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.727229    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:29:32.727358 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.727276    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:29:32.729661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.729642    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"]
Nov 01 07:29:32.803768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.803731    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.804023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.803784    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzgz\" (UniqueName: \"kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.804023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.803826    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.804524 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.804500    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.806147 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.806130    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.859851 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.859820    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzgz\" (UniqueName: \"kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz\") pod \"maas-api-revoke-sub-461f4fba-7w8vh\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.934069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:32.933998    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:32.952392 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:29:32.952357    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:29:33.306866 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:33.306839    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"]
Nov 01 07:29:33.309117 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:29:33.309080    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220bdd97_ae04_4a0d_9ad4_6e1da915146a.slice/crio-37f5d3ee377b89bd4eb7b71fdfcf2fcd5e19d061cbcc79aec87b1d69660e4b78 WatchSource:0}: Error finding container 37f5d3ee377b89bd4eb7b71fdfcf2fcd5e19d061cbcc79aec87b1d69660e4b78: Status 404 returned error can't find the container with id 37f5d3ee377b89bd4eb7b71fdfcf2fcd5e19d061cbcc79aec87b1d69660e4b78
Nov 01 07:29:33.523512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:33.523416    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh" event={"ID":"220bdd97-ae04-4a0d-9ad4-6e1da915146a","Type":"ContainerStarted","Data":"37f5d3ee377b89bd4eb7b71fdfcf2fcd5e19d061cbcc79aec87b1d69660e4b78"}
Nov 01 07:29:34.530227 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:34.530138    4468 generic.go:356] "Generic (PLEG): container finished" podID="220bdd97-ae04-4a0d-9ad4-6e1da915146a" containerID="21b82c6adc719d5b2e1fa0b5559fd82995b6e2229370ca350422ddd2cb955e06" exitCode=0
Nov 01 07:29:34.530633 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:34.530217    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh" event={"ID":"220bdd97-ae04-4a0d-9ad4-6e1da915146a","Type":"ContainerDied","Data":"21b82c6adc719d5b2e1fa0b5559fd82995b6e2229370ca350422ddd2cb955e06"}
Nov 01 07:29:35.668183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.668158    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:35.732994 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.732966    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz\" (UniqueName: \"kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz\") pod \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") "
Nov 01 07:29:35.733157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.733053    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca\") pod \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") "
Nov 01 07:29:35.733157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.733115    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca\") pod \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\" (UID: \"220bdd97-ae04-4a0d-9ad4-6e1da915146a\") "
Nov 01 07:29:35.733456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.733434    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca" pod "220bdd97-ae04-4a0d-9ad4-6e1da915146a" (UID: "220bdd97-ae04-4a0d-9ad4-6e1da915146a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:29:35.735201 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.735183    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca" pod "220bdd97-ae04-4a0d-9ad4-6e1da915146a" (UID: "220bdd97-ae04-4a0d-9ad4-6e1da915146a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:29:35.735287 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.735213    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz" pod "220bdd97-ae04-4a0d-9ad4-6e1da915146a" (UID: "220bdd97-ae04-4a0d-9ad4-6e1da915146a"). InnerVolumeSpecName "kube-api-access-7fzgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:29:35.834421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.834394    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fzgz\" (UniqueName: \"kubernetes.io/projected/220bdd97-ae04-4a0d-9ad4-6e1da915146a-kube-api-access-7fzgz\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:29:35.834421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.834421    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:29:35.834634 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:35.834432    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220bdd97-ae04-4a0d-9ad4-6e1da915146a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:29:36.541602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:36.541564    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh" event={"ID":"220bdd97-ae04-4a0d-9ad4-6e1da915146a","Type":"ContainerDied","Data":"37f5d3ee377b89bd4eb7b71fdfcf2fcd5e19d061cbcc79aec87b1d69660e4b78"}
Nov 01 07:29:36.541778 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:36.541607    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"
Nov 01 07:29:36.541778 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:36.541622    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f5d3ee377b89bd4eb7b71fdfcf2fcd5e19d061cbcc79aec87b1d69660e4b78"
Nov 01 07:29:43.044840 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:43.044796    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"]
Nov 01 07:29:43.094003 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:43.093970    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-r5bbb"]
Nov 01 07:29:44.588521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:44.588489    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f770f40e-e825-4643-acf7-fdc79a38dafc" path="/var/lib/kubelet/pods/f770f40e-e825-4643-acf7-fdc79a38dafc/volumes"
Nov 01 07:29:45.501217 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:29:45.501189    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:30:00.162035 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.161950    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"]
Nov 01 07:30:00.162602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.162586    4468 cpu_manager.go:414] "containerMap: removing container" podUID="220bdd97-ae04-4a0d-9ad4-6e1da915146a" containerName="revoke-keys"
Nov 01 07:30:00.162659 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.162604    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220bdd97-ae04-4a0d-9ad4-6e1da915146a" containerName="revoke-keys"
Nov 01 07:30:00.162706 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.162696    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="220bdd97-ae04-4a0d-9ad4-6e1da915146a" containerName="revoke-keys"
Nov 01 07:30:00.167379 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.167361    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.221910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.221864    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"]
Nov 01 07:30:00.268373 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.268342    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:30:00.268568 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.268489    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:30:00.269505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.269492    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:30:00.274044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.274019    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891970-rv4wp\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.274163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.274061    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sllm\" (UniqueName: \"kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm\") pod \"maas-api-key-cleanup-29891970-rv4wp\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.375825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.375790    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891970-rv4wp\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.376031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.375836    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sllm\" (UniqueName: \"kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm\") pod \"maas-api-key-cleanup-29891970-rv4wp\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.376350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.376329    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891970-rv4wp\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.435157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.435079    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sllm\" (UniqueName: \"kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm\") pod \"maas-api-key-cleanup-29891970-rv4wp\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.480169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.480141    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:00.849494 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:00.849452    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"]
Nov 01 07:30:00.851034 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:30:00.851011    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f7bd93_09a8_48ce_8013_80dfc047695d.slice/crio-4dde6c64267ef20932b3364d963487468250525a1cd19d718b6651c8c06dff9f WatchSource:0}: Error finding container 4dde6c64267ef20932b3364d963487468250525a1cd19d718b6651c8c06dff9f: Status 404 returned error can't find the container with id 4dde6c64267ef20932b3364d963487468250525a1cd19d718b6651c8c06dff9f
Nov 01 07:30:01.667138 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:01.667105    4468 generic.go:356] "Generic (PLEG): container finished" podID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerID="b117306c61e9b8979f5ab67bb471a59837fb82fe0531d5ed641fd105e37dbfca" exitCode=60
Nov 01 07:30:01.667589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:01.667184    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" event={"ID":"49f7bd93-09a8-48ce-8013-80dfc047695d","Type":"ContainerDied","Data":"b117306c61e9b8979f5ab67bb471a59837fb82fe0531d5ed641fd105e37dbfca"}
Nov 01 07:30:01.667589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:01.667228    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" event={"ID":"49f7bd93-09a8-48ce-8013-80dfc047695d","Type":"ContainerStarted","Data":"4dde6c64267ef20932b3364d963487468250525a1cd19d718b6651c8c06dff9f"}
Nov 01 07:30:01.667589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:01.667388    4468 scope.go:122] "RemoveContainer" containerID="b117306c61e9b8979f5ab67bb471a59837fb82fe0531d5ed641fd105e37dbfca"
Nov 01 07:30:02.673988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:02.673903    4468 generic.go:356] "Generic (PLEG): container finished" podID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerID="57aba32a8d624502ea4922c456fb88d79a2f2a884cff9b370e4fc2d0452fcc55" exitCode=60
Nov 01 07:30:02.674421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:02.673984    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" event={"ID":"49f7bd93-09a8-48ce-8013-80dfc047695d","Type":"ContainerDied","Data":"57aba32a8d624502ea4922c456fb88d79a2f2a884cff9b370e4fc2d0452fcc55"}
Nov 01 07:30:02.674421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:02.674042    4468 scope.go:122] "RemoveContainer" containerID="b117306c61e9b8979f5ab67bb471a59837fb82fe0531d5ed641fd105e37dbfca"
Nov 01 07:30:02.674421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:02.674266    4468 scope.go:122] "RemoveContainer" containerID="57aba32a8d624502ea4922c456fb88d79a2f2a884cff9b370e4fc2d0452fcc55"
Nov 01 07:30:02.674654 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:30:02.674549    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891970-rv4wp_odh-ai-gateway-infra(49f7bd93-09a8-48ce-8013-80dfc047695d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d"
Nov 01 07:30:02.947531 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:30:02.947425    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:30:03.681164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:03.681133    4468 scope.go:122] "RemoveContainer" containerID="57aba32a8d624502ea4922c456fb88d79a2f2a884cff9b370e4fc2d0452fcc55"
Nov 01 07:30:03.681641 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:30:03.681396    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891970-rv4wp_odh-ai-gateway-infra(49f7bd93-09a8-48ce-8013-80dfc047695d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d"
Nov 01 07:30:12.583036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:12.583004    4468 scope.go:122] "RemoveContainer" containerID="57aba32a8d624502ea4922c456fb88d79a2f2a884cff9b370e4fc2d0452fcc55"
Nov 01 07:30:13.732330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:13.732249    4468 generic.go:356] "Generic (PLEG): container finished" podID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerID="29ffa56d8cd86efa40798b23019fba038db442f8790cbb024c3c0b6c20e26c5e" exitCode=60
Nov 01 07:30:13.732330 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:13.732320    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" event={"ID":"49f7bd93-09a8-48ce-8013-80dfc047695d","Type":"ContainerDied","Data":"29ffa56d8cd86efa40798b23019fba038db442f8790cbb024c3c0b6c20e26c5e"}
Nov 01 07:30:13.732813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:13.732380    4468 scope.go:122] "RemoveContainer" containerID="57aba32a8d624502ea4922c456fb88d79a2f2a884cff9b370e4fc2d0452fcc55"
Nov 01 07:30:13.732813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:13.732692    4468 scope.go:122] "RemoveContainer" containerID="29ffa56d8cd86efa40798b23019fba038db442f8790cbb024c3c0b6c20e26c5e"
Nov 01 07:30:13.732945 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:30:13.732923    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891970-rv4wp_odh-ai-gateway-infra(49f7bd93-09a8-48ce-8013-80dfc047695d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d"
Nov 01 07:30:14.855859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:14.855830    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"]
Nov 01 07:30:14.991164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:14.991140    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:15.030263 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.030241    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca\") pod \"49f7bd93-09a8-48ce-8013-80dfc047695d\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") "
Nov 01 07:30:15.030404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.030328    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm\" (UniqueName: \"kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm\") pod \"49f7bd93-09a8-48ce-8013-80dfc047695d\" (UID: \"49f7bd93-09a8-48ce-8013-80dfc047695d\") "
Nov 01 07:30:15.030650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.030628    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca" pod "49f7bd93-09a8-48ce-8013-80dfc047695d" (UID: "49f7bd93-09a8-48ce-8013-80dfc047695d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:30:15.030766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.030752    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f7bd93-09a8-48ce-8013-80dfc047695d-maas-api-service-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:30:15.032416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.032391    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm" pod "49f7bd93-09a8-48ce-8013-80dfc047695d" (UID: "49f7bd93-09a8-48ce-8013-80dfc047695d"). InnerVolumeSpecName "kube-api-access-5sllm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:30:15.046788 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.046760    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"]
Nov 01 07:30:15.049297 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.049276    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-djw8s"]
Nov 01 07:30:15.131979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.131903    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sllm\" (UniqueName: \"kubernetes.io/projected/49f7bd93-09a8-48ce-8013-80dfc047695d-kube-api-access-5sllm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:30:15.744541 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.744514    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"
Nov 01 07:30:15.744704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.744516    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp" event={"ID":"49f7bd93-09a8-48ce-8013-80dfc047695d","Type":"ContainerDied","Data":"4dde6c64267ef20932b3364d963487468250525a1cd19d718b6651c8c06dff9f"}
Nov 01 07:30:15.744704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.744648    4468 scope.go:122] "RemoveContainer" containerID="29ffa56d8cd86efa40798b23019fba038db442f8790cbb024c3c0b6c20e26c5e"
Nov 01 07:30:15.906615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.906576    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"]
Nov 01 07:30:15.956169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:15.956135    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891970-rv4wp"]
Nov 01 07:30:16.589650 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:16.589616    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2140c7ac-a64e-4a9e-893d-1af0b992f2b8" path="/var/lib/kubelet/pods/2140c7ac-a64e-4a9e-893d-1af0b992f2b8/volumes"
Nov 01 07:30:16.589997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:16.589984    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" path="/var/lib/kubelet/pods/49f7bd93-09a8-48ce-8013-80dfc047695d/volumes"
Nov 01 07:30:32.949410 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:30:32.949372    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:30:34.050323 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:34.050281    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"]
Nov 01 07:30:34.099322 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:34.099285    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mx2h5"]
Nov 01 07:30:34.589022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:34.588988    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08532d8-767d-4417-a47a-22ae56cb2c1c" path="/var/lib/kubelet/pods/c08532d8-767d-4417-a47a-22ae56cb2c1c/volumes"
Nov 01 07:30:34.961969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:34.961894    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:30:47.093508 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:47.093439    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"]
Nov 01 07:30:47.095305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:47.095280    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p262b"]
Nov 01 07:30:48.589731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:48.589700    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db20bb44-9136-4e79-b62b-c040ff65fbfb" path="/var/lib/kubelet/pods/db20bb44-9136-4e79-b62b-c040ff65fbfb/volumes"
Nov 01 07:30:53.011456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:53.011421    4468 scope.go:122] "RemoveContainer" containerID="9253bb1b9a523bb644769797812d7bcdf6f512b6dac2f4ef5fe2256260205a50"
Nov 01 07:30:53.020496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:30:53.020459    4468 scope.go:122] "RemoveContainer" containerID="de71c28627954390bd960940b02313426d681185c51602d62e3e8a943dd692ae"
Nov 01 07:31:02.947389 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:31:02.947343    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:31:32.948351 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:31:32.948312    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:31:53.051450 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:31:53.051414    4468 scope.go:122] "RemoveContainer" containerID="765d7197ba4c35d78b417d096447f9178073673b79a4b3b57fe29d38831195e0"
Nov 01 07:31:53.060632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:31:53.060609    4468 scope.go:122] "RemoveContainer" containerID="29bcb14cbf07e291d397107aa977a503f8d1c9e4bd28037ba85df57c1256727e"
Nov 01 07:31:53.069491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:31:53.069447    4468 scope.go:122] "RemoveContainer" containerID="1616d040de2d63d3af7f2d2b2589a44332111000b3fc98a3ca9957b586ea9797"
Nov 01 07:32:01.044353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:01.044315    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"]
Nov 01 07:32:01.093952 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:01.093919    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e011430c-hckpq"]
Nov 01 07:32:02.589164 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:02.589130    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7362266b-1eaf-484e-b273-088f89d76f05" path="/var/lib/kubelet/pods/7362266b-1eaf-484e-b273-088f89d76f05/volumes"
Nov 01 07:32:02.948109 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:32:02.948014    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:32:15.467239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467203    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"]
Nov 01 07:32:15.467759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467687    4468 cpu_manager.go:414] "containerMap: removing container" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:15.467759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467700    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:15.467759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467730    4468 cpu_manager.go:414] "containerMap: removing container" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:15.467759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467736    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:15.467898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467789    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:15.467898 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.467801    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:15.471079 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.471061    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.563756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.563719    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkq6h\" (UniqueName: \"kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.563756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.563760    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.563989 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.563908    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.574277 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.574230    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:32:15.574428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.574284    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:32:15.574428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.574281    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:32:15.574428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.574333    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:32:15.576511 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.576489    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"]
Nov 01 07:32:15.664825 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.664791    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.665050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.664872    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkq6h\" (UniqueName: \"kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.665050 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.664898    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.665444 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.665419    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.667415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.667396    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.721455 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.721374    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkq6h\" (UniqueName: \"kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h\") pod \"maas-api-revoke-sub-e2547622-bllb6\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:15.784124 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:15.784085    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:16.156341 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:16.156318    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"]
Nov 01 07:32:16.158219 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:32:16.158192    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5106f047_8c43_4d70_a733_30c42d4abc7d.slice/crio-a9dba0c8811407e5b91c9bdbdd0a5a737ef13de8a30207576080ba2d61015924 WatchSource:0}: Error finding container a9dba0c8811407e5b91c9bdbdd0a5a737ef13de8a30207576080ba2d61015924: Status 404 returned error can't find the container with id a9dba0c8811407e5b91c9bdbdd0a5a737ef13de8a30207576080ba2d61015924
Nov 01 07:32:16.372984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:16.372946    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6" event={"ID":"5106f047-8c43-4d70-a733-30c42d4abc7d","Type":"ContainerStarted","Data":"a9dba0c8811407e5b91c9bdbdd0a5a737ef13de8a30207576080ba2d61015924"}
Nov 01 07:32:17.379813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:17.379721    4468 generic.go:356] "Generic (PLEG): container finished" podID="5106f047-8c43-4d70-a733-30c42d4abc7d" containerID="d462a3578b1506f4645c067d5cd289046bfc8a74be88212968ef33d370717a89" exitCode=0
Nov 01 07:32:17.379813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:17.379779    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6" event={"ID":"5106f047-8c43-4d70-a733-30c42d4abc7d","Type":"ContainerDied","Data":"d462a3578b1506f4645c067d5cd289046bfc8a74be88212968ef33d370717a89"}
Nov 01 07:32:18.523341 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.523312    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:18.595916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.595882    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca\") pod \"5106f047-8c43-4d70-a733-30c42d4abc7d\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") "
Nov 01 07:32:18.596093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.595945    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h\" (UniqueName: \"kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h\") pod \"5106f047-8c43-4d70-a733-30c42d4abc7d\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") "
Nov 01 07:32:18.596093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.596024    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca\") pod \"5106f047-8c43-4d70-a733-30c42d4abc7d\" (UID: \"5106f047-8c43-4d70-a733-30c42d4abc7d\") "
Nov 01 07:32:18.596338 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.596307    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca" pod "5106f047-8c43-4d70-a733-30c42d4abc7d" (UID: "5106f047-8c43-4d70-a733-30c42d4abc7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:32:18.596496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.596479    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:32:18.598244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.598222    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca" pod "5106f047-8c43-4d70-a733-30c42d4abc7d" (UID: "5106f047-8c43-4d70-a733-30c42d4abc7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:32:18.598337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.598227    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h" pod "5106f047-8c43-4d70-a733-30c42d4abc7d" (UID: "5106f047-8c43-4d70-a733-30c42d4abc7d"). InnerVolumeSpecName "kube-api-access-kkq6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:32:18.697545 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.697428    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5106f047-8c43-4d70-a733-30c42d4abc7d-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:32:18.697545 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:18.697503    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkq6h\" (UniqueName: \"kubernetes.io/projected/5106f047-8c43-4d70-a733-30c42d4abc7d-kube-api-access-kkq6h\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:32:19.392484 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:19.392442    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"
Nov 01 07:32:19.392661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:19.392445    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6" event={"ID":"5106f047-8c43-4d70-a733-30c42d4abc7d","Type":"ContainerDied","Data":"a9dba0c8811407e5b91c9bdbdd0a5a737ef13de8a30207576080ba2d61015924"}
Nov 01 07:32:19.392661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:19.392608    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9dba0c8811407e5b91c9bdbdd0a5a737ef13de8a30207576080ba2d61015924"
Nov 01 07:32:20.044220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:20.044188    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"]
Nov 01 07:32:20.093552 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:20.093515    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6e2041-hntbw"]
Nov 01 07:32:20.589609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:20.589573    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6766fee-93a4-4559-8669-f2d47140ecc4" path="/var/lib/kubelet/pods/b6766fee-93a4-4559-8669-f2d47140ecc4/volumes"
Nov 01 07:32:30.167300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:30.167268    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:32:32.947747 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:32:32.947709    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:32:40.507421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:40.507387    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:32:45.895148 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895115    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"]
Nov 01 07:32:45.895602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895551    4468 cpu_manager.go:414] "containerMap: removing container" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:45.895602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895565    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:45.895602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895576    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5106f047-8c43-4d70-a733-30c42d4abc7d" containerName="revoke-keys"
Nov 01 07:32:45.895602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895580    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5106f047-8c43-4d70-a733-30c42d4abc7d" containerName="revoke-keys"
Nov 01 07:32:45.895768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895665    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f7bd93-09a8-48ce-8013-80dfc047695d" containerName="cleanup"
Nov 01 07:32:45.895768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.895674    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5106f047-8c43-4d70-a733-30c42d4abc7d" containerName="revoke-keys"
Nov 01 07:32:45.898668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.898653    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:45.921312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.921283    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:45.921415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.921333    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6qbf\" (UniqueName: \"kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:45.921492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.921415    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:45.954130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:45.954108    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:32:46.003965 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.003931    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:32:46.004195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.003995    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:32:46.004195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.004170    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:32:46.006185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.006164    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"]
Nov 01 07:32:46.022528 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.022503    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.022669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.022612    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.022669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.022657    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6qbf\" (UniqueName: \"kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.023258 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.023237    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.025015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.024993    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.078957 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.078924    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6qbf\" (UniqueName: \"kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf\") pod \"maas-api-revoke-sub-c1e271a9-6rn8g\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.211194 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.211137    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:46.582084 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:46.582056    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"]
Nov 01 07:32:46.584370 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:32:46.584336    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdb334b_3ba9_4201_91d1_c8ccea8fd1b0.slice/crio-a10810f4499ab5dc95960f852c24036dcaac19b5c863422d61dd095b389adb49 WatchSource:0}: Error finding container a10810f4499ab5dc95960f852c24036dcaac19b5c863422d61dd095b389adb49: Status 404 returned error can't find the container with id a10810f4499ab5dc95960f852c24036dcaac19b5c863422d61dd095b389adb49
Nov 01 07:32:47.585574 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:47.585528    4468 generic.go:356] "Generic (PLEG): container finished" podID="6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" containerID="f144ca51c4a790831c4fc4f6654778a94ebc223335ef3b0c73de97ea693b7f44" exitCode=0
Nov 01 07:32:47.586039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:47.585593    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g" event={"ID":"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0","Type":"ContainerDied","Data":"f144ca51c4a790831c4fc4f6654778a94ebc223335ef3b0c73de97ea693b7f44"}
Nov 01 07:32:47.586039 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:47.585662    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g" event={"ID":"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0","Type":"ContainerStarted","Data":"a10810f4499ab5dc95960f852c24036dcaac19b5c863422d61dd095b389adb49"}
Nov 01 07:32:48.726770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.726747    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:48.850881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.850810    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca\") pod \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") "
Nov 01 07:32:48.850881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.850859    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf\" (UniqueName: \"kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf\") pod \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") "
Nov 01 07:32:48.851099 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.850938    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca\") pod \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\" (UID: \"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0\") "
Nov 01 07:32:48.851201 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.851176    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca" pod "6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" (UID: "6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:32:48.851381 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.851363    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:32:48.853113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.853088    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf" pod "6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" (UID: "6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0"). InnerVolumeSpecName "kube-api-access-v6qbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:32:48.853255 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.853138    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca" pod "6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" (UID: "6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:32:48.952409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.952385    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6qbf\" (UniqueName: \"kubernetes.io/projected/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-kube-api-access-v6qbf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:32:48.952409 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:48.952406    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:32:49.597430 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:49.597396    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"
Nov 01 07:32:49.597622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:49.597424    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g" event={"ID":"6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0","Type":"ContainerDied","Data":"a10810f4499ab5dc95960f852c24036dcaac19b5c863422d61dd095b389adb49"}
Nov 01 07:32:49.597622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:49.597499    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10810f4499ab5dc95960f852c24036dcaac19b5c863422d61dd095b389adb49"
Nov 01 07:32:58.401534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.401500    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"]
Nov 01 07:32:58.401977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.401913    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" containerName="revoke-keys"
Nov 01 07:32:58.401977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.401924    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" containerName="revoke-keys"
Nov 01 07:32:58.402053 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.402012    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" containerName="revoke-keys"
Nov 01 07:32:58.406274 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.406249    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.508822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.508779    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:32:58.509004 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.508849    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:32:58.509004 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.508873    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:32:58.509004 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.508929    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:32:58.511261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.511239    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"]
Nov 01 07:32:58.545548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.545507    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.545743 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.545637    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zm5w\" (UniqueName: \"kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.545743 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.545712    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.647208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.647171    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.647420 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.647247    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zm5w\" (UniqueName: \"kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.647420 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.647322    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.647823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.647802    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.649828 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.649806    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.703891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.703814    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zm5w\" (UniqueName: \"kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w\") pod \"maas-api-revoke-sub-ec843018-8h9g6\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.725666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.725630    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:32:58.770231 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:58.770198    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:32:59.098922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:59.098884    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"]
Nov 01 07:32:59.100934 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:32:59.100896    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a25fdc_38cf_418e_9574_bdb204bb0d6b.slice/crio-3e731bfb79a3d0f78a6dfc0e6209b22d516b767a387f3fa8754cc768285b465b WatchSource:0}: Error finding container 3e731bfb79a3d0f78a6dfc0e6209b22d516b767a387f3fa8754cc768285b465b: Status 404 returned error can't find the container with id 3e731bfb79a3d0f78a6dfc0e6209b22d516b767a387f3fa8754cc768285b465b
Nov 01 07:32:59.649173 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:59.649141    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6" event={"ID":"d5a25fdc-38cf-418e-9574-bdb204bb0d6b","Type":"ContainerStarted","Data":"a6d96c4cdc7645435d9fadb35b44d0b00df789cc788ef6e684778f1bb6fbf229"}
Nov 01 07:32:59.649669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:59.649194    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6" event={"ID":"d5a25fdc-38cf-418e-9574-bdb204bb0d6b","Type":"ContainerStarted","Data":"3e731bfb79a3d0f78a6dfc0e6209b22d516b767a387f3fa8754cc768285b465b"}
Nov 01 07:32:59.764517 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:32:59.764396    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6" podStartSLOduration=1.7643832229999998 podStartE2EDuration="1.764383223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:32:59.761624394 +0000 UTC m=+4991.329431228" watchObservedRunningTime="2026-11-01 07:32:59.764383223 +0000 UTC m=+4991.332190052"
Nov 01 07:33:00.662583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:00.662552    4468 generic.go:356] "Generic (PLEG): container finished" podID="d5a25fdc-38cf-418e-9574-bdb204bb0d6b" containerID="a6d96c4cdc7645435d9fadb35b44d0b00df789cc788ef6e684778f1bb6fbf229" exitCode=0
Nov 01 07:33:00.662997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:00.662595    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6" event={"ID":"d5a25fdc-38cf-418e-9574-bdb204bb0d6b","Type":"ContainerDied","Data":"a6d96c4cdc7645435d9fadb35b44d0b00df789cc788ef6e684778f1bb6fbf229"}
Nov 01 07:33:01.797052 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.797027    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:33:01.878976 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.878941    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca\") pod \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") "
Nov 01 07:33:01.879163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.878986    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w\" (UniqueName: \"kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w\") pod \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") "
Nov 01 07:33:01.879163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.879137    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca\") pod \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\" (UID: \"d5a25fdc-38cf-418e-9574-bdb204bb0d6b\") "
Nov 01 07:33:01.879512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.879489    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca" pod "d5a25fdc-38cf-418e-9574-bdb204bb0d6b" (UID: "d5a25fdc-38cf-418e-9574-bdb204bb0d6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:33:01.881198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.881179    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w" pod "d5a25fdc-38cf-418e-9574-bdb204bb0d6b" (UID: "d5a25fdc-38cf-418e-9574-bdb204bb0d6b"). InnerVolumeSpecName "kube-api-access-8zm5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:33:01.881251 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.881192    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca" pod "d5a25fdc-38cf-418e-9574-bdb204bb0d6b" (UID: "d5a25fdc-38cf-418e-9574-bdb204bb0d6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:33:01.980361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.980269    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:01.980361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.980303    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:01.980361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:01.980317    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zm5w\" (UniqueName: \"kubernetes.io/projected/d5a25fdc-38cf-418e-9574-bdb204bb0d6b-kube-api-access-8zm5w\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:02.674601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:02.674570    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"
Nov 01 07:33:02.674786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:02.674562    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6" event={"ID":"d5a25fdc-38cf-418e-9574-bdb204bb0d6b","Type":"ContainerDied","Data":"3e731bfb79a3d0f78a6dfc0e6209b22d516b767a387f3fa8754cc768285b465b"}
Nov 01 07:33:02.674786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:02.674692    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e731bfb79a3d0f78a6dfc0e6209b22d516b767a387f3fa8754cc768285b465b"
Nov 01 07:33:02.948672 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:33:02.948586    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:33:15.577948 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:15.577915    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:33:18.337492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:18.334191    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qb2mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:33:18.337492 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:18.334252    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qb2mm" podUID="a7adc820-4dbf-455b-ada4-9b655c77c786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:33:18.479656 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:18.479607    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hjj86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:33:18.479869 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:18.479683    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hjj86" podUID="8dfde8ce-8829-4310-a77e-0e64d77c0c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:33:32.945613 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:33:32.945575    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:33:33.515177 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.515148    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"]
Nov 01 07:33:33.515595 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.515581    4468 cpu_manager.go:414] "containerMap: removing container" podUID="d5a25fdc-38cf-418e-9574-bdb204bb0d6b" containerName="revoke-keys"
Nov 01 07:33:33.515655 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.515596    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a25fdc-38cf-418e-9574-bdb204bb0d6b" containerName="revoke-keys"
Nov 01 07:33:33.515704 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.515669    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a25fdc-38cf-418e-9574-bdb204bb0d6b" containerName="revoke-keys"
Nov 01 07:33:33.518620 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.518604    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.578594 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.578561    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"]
Nov 01 07:33:33.625638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.625601    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:33:33.625638 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.625612    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:33:33.625844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.625639    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:33:33.625844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.625679    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:33:33.627150 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.627129    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.627255 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.627242    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52k7\" (UniqueName: \"kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.627295 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.627288    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.728900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.728871    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.729057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.728921    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.729057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.728989    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52k7\" (UniqueName: \"kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.729525 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.729503    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.731383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.731366    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.785159 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.785093    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52k7\" (UniqueName: \"kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7\") pod \"maas-api-revoke-sub-046e1343-wdzpp\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:33.831019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:33.830986    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:34.199732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:34.199708    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"]
Nov 01 07:33:34.202002 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:33:34.201972    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756d76d9_67b9_4f9e_b3eb_2b39752357f1.slice/crio-7acdb0359a6c6fd68baa0ca74547f24ce664196145576f9267b1427ee1700cef WatchSource:0}: Error finding container 7acdb0359a6c6fd68baa0ca74547f24ce664196145576f9267b1427ee1700cef: Status 404 returned error can't find the container with id 7acdb0359a6c6fd68baa0ca74547f24ce664196145576f9267b1427ee1700cef
Nov 01 07:33:34.843759 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:34.843722    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp" event={"ID":"756d76d9-67b9-4f9e-b3eb-2b39752357f1","Type":"ContainerStarted","Data":"27fbce60971da4192961487cc218e384a6b0026b33d3ae6f7e5d37bc076aa33b"}
Nov 01 07:33:34.843942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:34.843777    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp" event={"ID":"756d76d9-67b9-4f9e-b3eb-2b39752357f1","Type":"ContainerStarted","Data":"7acdb0359a6c6fd68baa0ca74547f24ce664196145576f9267b1427ee1700cef"}
Nov 01 07:33:34.957049 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:34.956995    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp" podStartSLOduration=1.9569803399999999 podStartE2EDuration="1.95698034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:33:34.955886007 +0000 UTC m=+5026.523692850" watchObservedRunningTime="2026-11-01 07:33:34.95698034 +0000 UTC m=+5026.524787168"
Nov 01 07:33:35.850285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:35.850250    4468 generic.go:356] "Generic (PLEG): container finished" podID="756d76d9-67b9-4f9e-b3eb-2b39752357f1" containerID="27fbce60971da4192961487cc218e384a6b0026b33d3ae6f7e5d37bc076aa33b" exitCode=0
Nov 01 07:33:35.850686 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:35.850335    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp" event={"ID":"756d76d9-67b9-4f9e-b3eb-2b39752357f1","Type":"ContainerDied","Data":"27fbce60971da4192961487cc218e384a6b0026b33d3ae6f7e5d37bc076aa33b"}
Nov 01 07:33:36.994675 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:36.994648    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:37.059567 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.059541    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca\") pod \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") "
Nov 01 07:33:37.059716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.059590    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7\" (UniqueName: \"kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7\") pod \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") "
Nov 01 07:33:37.059800 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.059784    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca\") pod \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\" (UID: \"756d76d9-67b9-4f9e-b3eb-2b39752357f1\") "
Nov 01 07:33:37.059871 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.059840    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca" pod "756d76d9-67b9-4f9e-b3eb-2b39752357f1" (UID: "756d76d9-67b9-4f9e-b3eb-2b39752357f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:33:37.060081 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.060066    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:37.061781 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.061756    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7" pod "756d76d9-67b9-4f9e-b3eb-2b39752357f1" (UID: "756d76d9-67b9-4f9e-b3eb-2b39752357f1"). InnerVolumeSpecName "kube-api-access-z52k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:33:37.061885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.061795    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca" pod "756d76d9-67b9-4f9e-b3eb-2b39752357f1" (UID: "756d76d9-67b9-4f9e-b3eb-2b39752357f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:33:37.161664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.161589    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756d76d9-67b9-4f9e-b3eb-2b39752357f1-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:37.161664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.161613    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z52k7\" (UniqueName: \"kubernetes.io/projected/756d76d9-67b9-4f9e-b3eb-2b39752357f1-kube-api-access-z52k7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:37.861658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.861624    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"
Nov 01 07:33:37.861658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.861637    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp" event={"ID":"756d76d9-67b9-4f9e-b3eb-2b39752357f1","Type":"ContainerDied","Data":"7acdb0359a6c6fd68baa0ca74547f24ce664196145576f9267b1427ee1700cef"}
Nov 01 07:33:37.861929 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:37.861685    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acdb0359a6c6fd68baa0ca74547f24ce664196145576f9267b1427ee1700cef"
Nov 01 07:33:42.044556 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:42.044518    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"]
Nov 01 07:33:42.094091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:42.094063    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9b6zz"]
Nov 01 07:33:42.589148 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:42.589116    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a4ccc0-83c7-4bd4-b225-1d819c66f1a5" path="/var/lib/kubelet/pods/00a4ccc0-83c7-4bd4-b225-1d819c66f1a5/volumes"
Nov 01 07:33:44.304300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.304266    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"]
Nov 01 07:33:44.304918 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.304898    4468 cpu_manager.go:414] "containerMap: removing container" podUID="756d76d9-67b9-4f9e-b3eb-2b39752357f1" containerName="revoke-keys"
Nov 01 07:33:44.305015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.304920    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756d76d9-67b9-4f9e-b3eb-2b39752357f1" containerName="revoke-keys"
Nov 01 07:33:44.305072 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.305037    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="756d76d9-67b9-4f9e-b3eb-2b39752357f1" containerName="revoke-keys"
Nov 01 07:33:44.308395 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.308374    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.410960 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.410926    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:33:44.411141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.410965    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:33:44.411141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.410966    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:33:44.411141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.411014    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:33:44.413168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.413149    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"]
Nov 01 07:33:44.436521 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.436495    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.436643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.436523    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.436684 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.436643    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27tb\" (UniqueName: \"kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.537334 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.537301    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.537534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.537341    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.537534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.537416    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27tb\" (UniqueName: \"kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.538046 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.538026    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.539824 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.539802    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.594040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.593976    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27tb\" (UniqueName: \"kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb\") pod \"maas-api-revoke-sub-d3fba056-86n6z\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.620588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.620559    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:44.988954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:44.988932    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"]
Nov 01 07:33:44.991814 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:33:44.991785    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334fade6_bcfa_4fe1_ad1d_318efa3c63eb.slice/crio-b6a14dfcb2861ca58d1c4d7808e0e409ec2d8df2e26352ab001cbe3bed6671db WatchSource:0}: Error finding container b6a14dfcb2861ca58d1c4d7808e0e409ec2d8df2e26352ab001cbe3bed6671db: Status 404 returned error can't find the container with id b6a14dfcb2861ca58d1c4d7808e0e409ec2d8df2e26352ab001cbe3bed6671db
Nov 01 07:33:45.901452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:45.901411    4468 generic.go:356] "Generic (PLEG): container finished" podID="334fade6-bcfa-4fe1-ad1d-318efa3c63eb" containerID="877f2d0bff30f3129bd59053db02144a16ee69d9c12dbce84d301d2556f5287c" exitCode=0
Nov 01 07:33:45.901864 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:45.901500    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z" event={"ID":"334fade6-bcfa-4fe1-ad1d-318efa3c63eb","Type":"ContainerDied","Data":"877f2d0bff30f3129bd59053db02144a16ee69d9c12dbce84d301d2556f5287c"}
Nov 01 07:33:45.901864 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:45.901580    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z" event={"ID":"334fade6-bcfa-4fe1-ad1d-318efa3c63eb","Type":"ContainerStarted","Data":"b6a14dfcb2861ca58d1c4d7808e0e409ec2d8df2e26352ab001cbe3bed6671db"}
Nov 01 07:33:46.391781 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:46.391747    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:33:47.075460 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.075433    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:47.164892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.164798    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb\" (UniqueName: \"kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb\") pod \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") "
Nov 01 07:33:47.164892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.164850    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca\") pod \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") "
Nov 01 07:33:47.165152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.164957    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca\") pod \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\" (UID: \"334fade6-bcfa-4fe1-ad1d-318efa3c63eb\") "
Nov 01 07:33:47.165560 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.165536    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca" pod "334fade6-bcfa-4fe1-ad1d-318efa3c63eb" (UID: "334fade6-bcfa-4fe1-ad1d-318efa3c63eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:33:47.167331 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.167309    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca" pod "334fade6-bcfa-4fe1-ad1d-318efa3c63eb" (UID: "334fade6-bcfa-4fe1-ad1d-318efa3c63eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:33:47.167628 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.167602    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb" pod "334fade6-bcfa-4fe1-ad1d-318efa3c63eb" (UID: "334fade6-bcfa-4fe1-ad1d-318efa3c63eb"). InnerVolumeSpecName "kube-api-access-r27tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:33:47.267061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.267021    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:47.267061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.267060    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27tb\" (UniqueName: \"kubernetes.io/projected/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-kube-api-access-r27tb\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:47.267261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.267077    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fade6-bcfa-4fe1-ad1d-318efa3c63eb-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:33:47.932482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.932426    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z" event={"ID":"334fade6-bcfa-4fe1-ad1d-318efa3c63eb","Type":"ContainerDied","Data":"b6a14dfcb2861ca58d1c4d7808e0e409ec2d8df2e26352ab001cbe3bed6671db"}
Nov 01 07:33:47.932791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.932772    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a14dfcb2861ca58d1c4d7808e0e409ec2d8df2e26352ab001cbe3bed6671db"
Nov 01 07:33:47.932922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:47.932664    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"
Nov 01 07:33:53.105968 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:53.105938    4468 scope.go:122] "RemoveContainer" containerID="ebf6f8802efda8703c4dfde9778d20182d91ae37cd88544fe764ad690dff7d04"
Nov 01 07:33:53.114380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:53.114360    4468 scope.go:122] "RemoveContainer" containerID="a6b0daf23347e0346ae24bdf690e70bfeac664b58c2c1de84d9e6403251a2fee"
Nov 01 07:33:56.996354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:33:56.996324    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:34:02.946442 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:34:02.946408    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:34:09.111360 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:09.111327    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:34:11.090241 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:11.090206    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"]
Nov 01 07:34:11.093796 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:11.093767    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2jqwt"]
Nov 01 07:34:12.589231 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:12.589200    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049949da-bcb1-44fd-beef-a88164bbbf8a" path="/var/lib/kubelet/pods/049949da-bcb1-44fd-beef-a88164bbbf8a/volumes"
Nov 01 07:34:19.041145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.041112    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"]
Nov 01 07:34:19.041683 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.041555    4468 cpu_manager.go:414] "containerMap: removing container" podUID="334fade6-bcfa-4fe1-ad1d-318efa3c63eb" containerName="revoke-keys"
Nov 01 07:34:19.041683 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.041568    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334fade6-bcfa-4fe1-ad1d-318efa3c63eb" containerName="revoke-keys"
Nov 01 07:34:19.041683 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.041655    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="334fade6-bcfa-4fe1-ad1d-318efa3c63eb" containerName="revoke-keys"
Nov 01 07:34:19.045859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.045843    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.101108 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.101078    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"]
Nov 01 07:34:19.147843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.147815    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:34:19.147843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.147836    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:34:19.148060 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.147860    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:34:19.148060 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.147876    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:34:19.192232 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.192207    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.192361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.192251    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.192361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.192316    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jh5m\" (UniqueName: \"kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.292940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.292871    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.292940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.292912    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.293131 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.292953    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jh5m\" (UniqueName: \"kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.293502 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.293482    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.295349 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.295331    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.348926 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.348901    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jh5m\" (UniqueName: \"kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m\") pod \"maas-api-revoke-sub-ae594979-49jjf\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.357888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.357871    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:19.731864 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:19.731842    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"]
Nov 01 07:34:19.733779 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:34:19.733748    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80da0a69_7c04_4e4f_b07f_aafa7e11bf7f.slice/crio-89f8e435f69bcdd2ee945958154f576fd75e310f75399cfccd65e10be2c1a64b WatchSource:0}: Error finding container 89f8e435f69bcdd2ee945958154f576fd75e310f75399cfccd65e10be2c1a64b: Status 404 returned error can't find the container with id 89f8e435f69bcdd2ee945958154f576fd75e310f75399cfccd65e10be2c1a64b
Nov 01 07:34:20.125158 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:20.125123    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf" event={"ID":"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f","Type":"ContainerStarted","Data":"ca4da3abb618ce205c706096784ece775e217a0fb7b51e588e406bb98738ef13"}
Nov 01 07:34:20.125540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:20.125185    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf" event={"ID":"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f","Type":"ContainerStarted","Data":"89f8e435f69bcdd2ee945958154f576fd75e310f75399cfccd65e10be2c1a64b"}
Nov 01 07:34:20.238785 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:20.238737    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf" podStartSLOduration=1.2387240720000001 podStartE2EDuration="1.238724072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:34:20.236770108 +0000 UTC m=+5071.804576938" watchObservedRunningTime="2026-11-01 07:34:20.238724072 +0000 UTC m=+5071.806530900"
Nov 01 07:34:21.130923 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:21.130892    4468 generic.go:356] "Generic (PLEG): container finished" podID="80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" containerID="ca4da3abb618ce205c706096784ece775e217a0fb7b51e588e406bb98738ef13" exitCode=0
Nov 01 07:34:21.131374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:21.130965    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf" event={"ID":"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f","Type":"ContainerDied","Data":"ca4da3abb618ce205c706096784ece775e217a0fb7b51e588e406bb98738ef13"}
Nov 01 07:34:22.270965 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.270943    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:22.426718 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.426649    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m\" (UniqueName: \"kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m\") pod \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") "
Nov 01 07:34:22.426900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.426730    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca\") pod \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") "
Nov 01 07:34:22.426900 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.426754    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca\") pod \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\" (UID: \"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f\") "
Nov 01 07:34:22.427127 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.427096    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca" pod "80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" (UID: "80da0a69-7c04-4e4f-b07f-aafa7e11bf7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:34:22.428872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.428851    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca" pod "80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" (UID: "80da0a69-7c04-4e4f-b07f-aafa7e11bf7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:34:22.428996 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.428978    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m" pod "80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" (UID: "80da0a69-7c04-4e4f-b07f-aafa7e11bf7f"). InnerVolumeSpecName "kube-api-access-5jh5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:34:22.528305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.528278    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jh5m\" (UniqueName: \"kubernetes.io/projected/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-kube-api-access-5jh5m\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:34:22.528305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.528300    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:34:22.528305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:22.528309    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:34:23.141259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:23.141227    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"
Nov 01 07:34:23.141440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:23.141275    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf" event={"ID":"80da0a69-7c04-4e4f-b07f-aafa7e11bf7f","Type":"ContainerDied","Data":"89f8e435f69bcdd2ee945958154f576fd75e310f75399cfccd65e10be2c1a64b"}
Nov 01 07:34:23.141440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:23.141319    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89f8e435f69bcdd2ee945958154f576fd75e310f75399cfccd65e10be2c1a64b"
Nov 01 07:34:31.885708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:31.885630    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:34:32.946821 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:34:32.946786    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:34:36.048272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:36.048235    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"]
Nov 01 07:34:36.097703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:36.097675    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f4fba-7w8vh"]
Nov 01 07:34:36.588640 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:36.588608    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220bdd97-ae04-4a0d-9ad4-6e1da915146a" path="/var/lib/kubelet/pods/220bdd97-ae04-4a0d-9ad4-6e1da915146a/volumes"
Nov 01 07:34:37.712371 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.712337    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"]
Nov 01 07:34:37.720512 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.718298    4468 cpu_manager.go:414] "containerMap: removing container" podUID="80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" containerName="revoke-keys"
Nov 01 07:34:37.720819 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.720801    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" containerName="revoke-keys"
Nov 01 07:34:37.721094 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.721077    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" containerName="revoke-keys"
Nov 01 07:34:37.732746 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.732728    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.778644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.777363    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"]
Nov 01 07:34:37.778644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.778275    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.778644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.778395    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.778644 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.778434    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8nx\" (UniqueName: \"kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.824967 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.824789    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:34:37.824967 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.824853    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:34:37.824967 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.824939    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:34:37.834626 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.834587    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:34:37.881376 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.881342    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.881619 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.881517    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.881877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.881857    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.882031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.882013    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8nx\" (UniqueName: \"kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.885557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.885535    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:37.955402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:37.955376    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8nx\" (UniqueName: \"kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx\") pod \"maas-api-revoke-sub-a4aef7d7-qbjx5\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:38.064490 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:38.061637    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:38.579267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:38.566255    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"]
Nov 01 07:34:38.579267 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:34:38.576486    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311da33e_8a35_4489_9054_b99c200c22cc.slice/crio-346659cfe3b7bbf3df709530953709f1e9ba209532cd787872477ba3bd2fb71f WatchSource:0}: Error finding container 346659cfe3b7bbf3df709530953709f1e9ba209532cd787872477ba3bd2fb71f: Status 404 returned error can't find the container with id 346659cfe3b7bbf3df709530953709f1e9ba209532cd787872477ba3bd2fb71f
Nov 01 07:34:39.292590 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:39.292506    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5" event={"ID":"311da33e-8a35-4489-9054-b99c200c22cc","Type":"ContainerStarted","Data":"e63b4d347aaf99dfe00350dd6de907a2f6ac4172f3212530ba990e98e1d28a4a"}
Nov 01 07:34:39.292590 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:39.292576    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5" event={"ID":"311da33e-8a35-4489-9054-b99c200c22cc","Type":"ContainerStarted","Data":"346659cfe3b7bbf3df709530953709f1e9ba209532cd787872477ba3bd2fb71f"}
Nov 01 07:34:39.406034 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:39.405982    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5" podStartSLOduration=2.405968962 podStartE2EDuration="2.405968962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:34:39.404178464 +0000 UTC m=+5090.971985316" watchObservedRunningTime="2026-11-01 07:34:39.405968962 +0000 UTC m=+5090.973775809"
Nov 01 07:34:40.299113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:40.299079    4468 generic.go:356] "Generic (PLEG): container finished" podID="311da33e-8a35-4489-9054-b99c200c22cc" containerID="e63b4d347aaf99dfe00350dd6de907a2f6ac4172f3212530ba990e98e1d28a4a" exitCode=0
Nov 01 07:34:40.299518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:40.299122    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5" event={"ID":"311da33e-8a35-4489-9054-b99c200c22cc","Type":"ContainerDied","Data":"e63b4d347aaf99dfe00350dd6de907a2f6ac4172f3212530ba990e98e1d28a4a"}
Nov 01 07:34:41.206245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.206207    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:34:41.434780 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.434757    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:41.514865 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.514811    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx\" (UniqueName: \"kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx\") pod \"311da33e-8a35-4489-9054-b99c200c22cc\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") "
Nov 01 07:34:41.514994 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.514880    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca\") pod \"311da33e-8a35-4489-9054-b99c200c22cc\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") "
Nov 01 07:34:41.514994 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.514917    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca\") pod \"311da33e-8a35-4489-9054-b99c200c22cc\" (UID: \"311da33e-8a35-4489-9054-b99c200c22cc\") "
Nov 01 07:34:41.515326 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.515302    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca" pod "311da33e-8a35-4489-9054-b99c200c22cc" (UID: "311da33e-8a35-4489-9054-b99c200c22cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:34:41.517014 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.516992    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca" pod "311da33e-8a35-4489-9054-b99c200c22cc" (UID: "311da33e-8a35-4489-9054-b99c200c22cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:34:41.517244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.517221    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx" pod "311da33e-8a35-4489-9054-b99c200c22cc" (UID: "311da33e-8a35-4489-9054-b99c200c22cc"). InnerVolumeSpecName "kube-api-access-xj8nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:34:41.616027 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.615993    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8nx\" (UniqueName: \"kubernetes.io/projected/311da33e-8a35-4489-9054-b99c200c22cc-kube-api-access-xj8nx\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:34:41.616027 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.616027    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:34:41.616208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:41.616043    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311da33e-8a35-4489-9054-b99c200c22cc-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:34:42.310019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:42.309986    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5" event={"ID":"311da33e-8a35-4489-9054-b99c200c22cc","Type":"ContainerDied","Data":"346659cfe3b7bbf3df709530953709f1e9ba209532cd787872477ba3bd2fb71f"}
Nov 01 07:34:42.310192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:42.310037    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346659cfe3b7bbf3df709530953709f1e9ba209532cd787872477ba3bd2fb71f"
Nov 01 07:34:42.310192 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:42.310000    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"
Nov 01 07:34:49.501922 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:49.501890    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:34:53.137009 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:53.136976    4468 scope.go:122] "RemoveContainer" containerID="13eac24d722d86776fc9bef588f297e12738e856e88567d7bc708ffcecc4f46a"
Nov 01 07:34:58.184028 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.183997    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:34:58.784400 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.784358    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"]
Nov 01 07:34:58.785036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.785014    4468 cpu_manager.go:414] "containerMap: removing container" podUID="311da33e-8a35-4489-9054-b99c200c22cc" containerName="revoke-keys"
Nov 01 07:34:58.785100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.785042    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311da33e-8a35-4489-9054-b99c200c22cc" containerName="revoke-keys"
Nov 01 07:34:58.785189 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.785176    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="311da33e-8a35-4489-9054-b99c200c22cc" containerName="revoke-keys"
Nov 01 07:34:58.790137 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.790112    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.844679 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.844647    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"]
Nov 01 07:34:58.876887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.876850    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.877082 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.876939    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfbd\" (UniqueName: \"kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.877153 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.877094    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.890351 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.890323    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:34:58.890538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.890373    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:34:58.891976 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.891953    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:34:58.892091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.892009    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:34:58.978717 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.978674    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.978908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.978769    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfbd\" (UniqueName: \"kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.978908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.978847    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.979430 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.979405    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:58.981449 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:58.981421    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:59.034857 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:59.034790    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfbd\" (UniqueName: \"kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd\") pod \"maas-api-revoke-sub-79805ca9-cx26c\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:59.106563 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:59.106512    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:34:59.500529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:34:59.500500    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"]
Nov 01 07:34:59.502188 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:34:59.502154    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda717f1e7_47e0_4429_ae32_cc71af3044dc.slice/crio-c1bc93813b42db4c3b961493db0bdbceceffbcefceb93b6bdf39e67ea0eaca6e WatchSource:0}: Error finding container c1bc93813b42db4c3b961493db0bdbceceffbcefceb93b6bdf39e67ea0eaca6e: Status 404 returned error can't find the container with id c1bc93813b42db4c3b961493db0bdbceceffbcefceb93b6bdf39e67ea0eaca6e
Nov 01 07:35:00.491497 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:00.491438    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c" event={"ID":"a717f1e7-47e0-4429-ae32-cc71af3044dc","Type":"ContainerStarted","Data":"ed9187fba29a024dca29dabc45689f49106e1614fdfcbc2fc4b5867236d65969"}
Nov 01 07:35:00.491677 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:00.491507    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c" event={"ID":"a717f1e7-47e0-4429-ae32-cc71af3044dc","Type":"ContainerStarted","Data":"c1bc93813b42db4c3b961493db0bdbceceffbcefceb93b6bdf39e67ea0eaca6e"}
Nov 01 07:35:00.634548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:00.633731    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c" podStartSLOduration=2.633695996 podStartE2EDuration="2.633695996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:35:00.629918437 +0000 UTC m=+5112.197725269" watchObservedRunningTime="2026-11-01 07:35:00.633695996 +0000 UTC m=+5112.201502826"
Nov 01 07:35:00.819488 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:00.816620    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:35:00.819488 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:00.816682    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:35:01.497549 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:01.497516    4468 generic.go:356] "Generic (PLEG): container finished" podID="a717f1e7-47e0-4429-ae32-cc71af3044dc" containerID="ed9187fba29a024dca29dabc45689f49106e1614fdfcbc2fc4b5867236d65969" exitCode=0
Nov 01 07:35:01.497760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:01.497565    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c" event={"ID":"a717f1e7-47e0-4429-ae32-cc71af3044dc","Type":"ContainerDied","Data":"ed9187fba29a024dca29dabc45689f49106e1614fdfcbc2fc4b5867236d65969"}
Nov 01 07:35:02.631766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.631742    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:35:02.716908 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.716880    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca\") pod \"a717f1e7-47e0-4429-ae32-cc71af3044dc\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") "
Nov 01 07:35:02.717073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.717029    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca\") pod \"a717f1e7-47e0-4429-ae32-cc71af3044dc\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") "
Nov 01 07:35:02.717145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.717083    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd\" (UniqueName: \"kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd\") pod \"a717f1e7-47e0-4429-ae32-cc71af3044dc\" (UID: \"a717f1e7-47e0-4429-ae32-cc71af3044dc\") "
Nov 01 07:35:02.717403 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.717379    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca" pod "a717f1e7-47e0-4429-ae32-cc71af3044dc" (UID: "a717f1e7-47e0-4429-ae32-cc71af3044dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:35:02.719069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.719039    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca" pod "a717f1e7-47e0-4429-ae32-cc71af3044dc" (UID: "a717f1e7-47e0-4429-ae32-cc71af3044dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:35:02.719182 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.719164    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd" pod "a717f1e7-47e0-4429-ae32-cc71af3044dc" (UID: "a717f1e7-47e0-4429-ae32-cc71af3044dc"). InnerVolumeSpecName "kube-api-access-fvfbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:35:02.818786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.818755    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfbd\" (UniqueName: \"kubernetes.io/projected/a717f1e7-47e0-4429-ae32-cc71af3044dc-kube-api-access-fvfbd\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:02.818786 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.818781    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:02.818981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:02.818791    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a717f1e7-47e0-4429-ae32-cc71af3044dc-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:02.946089 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:35:02.946055    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:35:03.509498 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:03.509454    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"
Nov 01 07:35:03.509708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:03.509498    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c" event={"ID":"a717f1e7-47e0-4429-ae32-cc71af3044dc","Type":"ContainerDied","Data":"c1bc93813b42db4c3b961493db0bdbceceffbcefceb93b6bdf39e67ea0eaca6e"}
Nov 01 07:35:03.509708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:03.509545    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1bc93813b42db4c3b961493db0bdbceceffbcefceb93b6bdf39e67ea0eaca6e"
Nov 01 07:35:07.194654 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:07.194619    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:35:15.823652 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:15.823596    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:35:15.824344 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:15.824297    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:35:16.208884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.208788    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"]
Nov 01 07:35:16.209798 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.209772    4468 cpu_manager.go:414] "containerMap: removing container" podUID="a717f1e7-47e0-4429-ae32-cc71af3044dc" containerName="revoke-keys"
Nov 01 07:35:16.209963 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.209948    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a717f1e7-47e0-4429-ae32-cc71af3044dc" containerName="revoke-keys"
Nov 01 07:35:16.210246 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.210228    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="a717f1e7-47e0-4429-ae32-cc71af3044dc" containerName="revoke-keys"
Nov 01 07:35:16.230547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.228879    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.284559 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.284530    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"]
Nov 01 07:35:16.308797 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.308745    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:35:16.327691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.326845    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:35:16.327691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.327568    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:35:16.327691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.327620    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:35:16.392179 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.392139    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnhr\" (UniqueName: \"kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.392412 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.392202    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.392412 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.392359    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.494121 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.494032    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.494282 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.494125    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnhr\" (UniqueName: \"kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.494282 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.494161    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.494784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.494755    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.496582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.496564    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.553002 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.552956    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnhr\" (UniqueName: \"kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr\") pod \"maas-api-revoke-sub-0b7e8edb-q8qfp\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.589967 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.589939    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:16.959744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:16.959719    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"]
Nov 01 07:35:16.961280 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:35:16.961252    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f7ab1e_4764_4e87_a906_685cd6e02265.slice/crio-6a967120fb2336fc63d492b8347378ff2b4593dc816ba055ac7eb66de6e605ee WatchSource:0}: Error finding container 6a967120fb2336fc63d492b8347378ff2b4593dc816ba055ac7eb66de6e605ee: Status 404 returned error can't find the container with id 6a967120fb2336fc63d492b8347378ff2b4593dc816ba055ac7eb66de6e605ee
Nov 01 07:35:17.603184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:17.603143    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp" event={"ID":"38f7ab1e-4764-4e87-a906-685cd6e02265","Type":"ContainerStarted","Data":"cd9d030a138c37ef6064192adefd7ed45748be730ad7ecb2f81b5861be5ea76c"}
Nov 01 07:35:17.603380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:17.603246    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp" event={"ID":"38f7ab1e-4764-4e87-a906-685cd6e02265","Type":"ContainerStarted","Data":"6a967120fb2336fc63d492b8347378ff2b4593dc816ba055ac7eb66de6e605ee"}
Nov 01 07:35:18.609363 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:18.609326    4468 generic.go:356] "Generic (PLEG): container finished" podID="38f7ab1e-4764-4e87-a906-685cd6e02265" containerID="cd9d030a138c37ef6064192adefd7ed45748be730ad7ecb2f81b5861be5ea76c" exitCode=0
Nov 01 07:35:18.609783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:18.609391    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp" event={"ID":"38f7ab1e-4764-4e87-a906-685cd6e02265","Type":"ContainerDied","Data":"cd9d030a138c37ef6064192adefd7ed45748be730ad7ecb2f81b5861be5ea76c"}
Nov 01 07:35:19.470416 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.470384    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:35:19.751822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.751794    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:19.825801 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.825767    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca\") pod \"38f7ab1e-4764-4e87-a906-685cd6e02265\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") "
Nov 01 07:35:19.826007 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.825806    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr\" (UniqueName: \"kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr\") pod \"38f7ab1e-4764-4e87-a906-685cd6e02265\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") "
Nov 01 07:35:19.826007 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.825881    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca\") pod \"38f7ab1e-4764-4e87-a906-685cd6e02265\" (UID: \"38f7ab1e-4764-4e87-a906-685cd6e02265\") "
Nov 01 07:35:19.826261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.826239    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca" pod "38f7ab1e-4764-4e87-a906-685cd6e02265" (UID: "38f7ab1e-4764-4e87-a906-685cd6e02265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:35:19.827890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.827873    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca" pod "38f7ab1e-4764-4e87-a906-685cd6e02265" (UID: "38f7ab1e-4764-4e87-a906-685cd6e02265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:35:19.828023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.828004    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr" pod "38f7ab1e-4764-4e87-a906-685cd6e02265" (UID: "38f7ab1e-4764-4e87-a906-685cd6e02265"). InnerVolumeSpecName "kube-api-access-qcnhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:35:19.926862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.926822    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:19.926862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.926856    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f7ab1e-4764-4e87-a906-685cd6e02265-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:19.926862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:19.926865    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnhr\" (UniqueName: \"kubernetes.io/projected/38f7ab1e-4764-4e87-a906-685cd6e02265-kube-api-access-qcnhr\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:20.621516 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:20.621483    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp" event={"ID":"38f7ab1e-4764-4e87-a906-685cd6e02265","Type":"ContainerDied","Data":"6a967120fb2336fc63d492b8347378ff2b4593dc816ba055ac7eb66de6e605ee"}
Nov 01 07:35:20.621658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:20.621531    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a967120fb2336fc63d492b8347378ff2b4593dc816ba055ac7eb66de6e605ee"
Nov 01 07:35:20.621658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:20.621496    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"
Nov 01 07:35:27.491018 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.490983    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"]
Nov 01 07:35:27.491485 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.491438    4468 cpu_manager.go:414] "containerMap: removing container" podUID="38f7ab1e-4764-4e87-a906-685cd6e02265" containerName="revoke-keys"
Nov 01 07:35:27.491485 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.491454    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f7ab1e-4764-4e87-a906-685cd6e02265" containerName="revoke-keys"
Nov 01 07:35:27.491582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.491570    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f7ab1e-4764-4e87-a906-685cd6e02265" containerName="revoke-keys"
Nov 01 07:35:27.494867 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.494849    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.553066 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.552838    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"]
Nov 01 07:35:27.597774 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.597738    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:35:27.597974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.597816    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:35:27.598069 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.598050    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:35:27.598195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.598175    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:35:27.608092 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.608058    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsk5\" (UniqueName: \"kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.608537 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.608514    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.608868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.608849    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.709767 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.709733    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.709928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.709791    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsk5\" (UniqueName: \"kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.709928 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.709820    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.710630 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.710607    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.712306 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.712273    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.767252 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.767165    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsk5\" (UniqueName: \"kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5\") pod \"maas-api-revoke-sub-3d4156f1-v2drp\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:27.803730 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.803690    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:35:27.808434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:27.808405    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:28.188310 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:28.188279    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"]
Nov 01 07:35:28.663359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:28.663326    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp" event={"ID":"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5","Type":"ContainerStarted","Data":"c27af9a903e1092036c1a3ad3ac1e2071655e3855d97d1c4f5f1db2d236954fd"}
Nov 01 07:35:28.663872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:28.663383    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp" event={"ID":"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5","Type":"ContainerStarted","Data":"80cd193cad5866a17eb51a8265de18c738c7bb0de76e5b4b940140043b9ef5b8"}
Nov 01 07:35:28.776841 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:28.776726    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp" podStartSLOduration=1.776713379 podStartE2EDuration="1.776713379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:35:28.775395476 +0000 UTC m=+5140.343202313" watchObservedRunningTime="2026-11-01 07:35:28.776713379 +0000 UTC m=+5140.344520208"
Nov 01 07:35:29.673809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:29.673777    4468 generic.go:356] "Generic (PLEG): container finished" podID="c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" containerID="c27af9a903e1092036c1a3ad3ac1e2071655e3855d97d1c4f5f1db2d236954fd" exitCode=0
Nov 01 07:35:29.674276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:29.673815    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp" event={"ID":"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5","Type":"ContainerDied","Data":"c27af9a903e1092036c1a3ad3ac1e2071655e3855d97d1c4f5f1db2d236954fd"}
Nov 01 07:35:30.820194 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.820168    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:30.941129 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.941040    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5\" (UniqueName: \"kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5\") pod \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") "
Nov 01 07:35:30.941320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.941143    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca\") pod \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") "
Nov 01 07:35:30.941320 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.941247    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca\") pod \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\" (UID: \"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5\") "
Nov 01 07:35:30.941547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.941518    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca" pod "c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" (UID: "c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:35:30.941809 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.941787    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:30.943903 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.943860    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca" pod "c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" (UID: "c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:35:30.944036 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:30.943918    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5" pod "c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" (UID: "c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5"). InnerVolumeSpecName "kube-api-access-kvsk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:35:31.042761 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:31.042714    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:31.042761 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:31.042757    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvsk5\" (UniqueName: \"kubernetes.io/projected/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5-kube-api-access-kvsk5\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:31.685611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:31.685577    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp" event={"ID":"c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5","Type":"ContainerDied","Data":"80cd193cad5866a17eb51a8265de18c738c7bb0de76e5b4b940140043b9ef5b8"}
Nov 01 07:35:31.685611 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:31.685599    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"
Nov 01 07:35:31.685843 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:31.685622    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80cd193cad5866a17eb51a8265de18c738c7bb0de76e5b4b940140043b9ef5b8"
Nov 01 07:35:32.947230 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:35:32.947188    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:35:42.008073 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:42.008035    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:35:45.628754 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.628714    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"]
Nov 01 07:35:45.678142 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.678107    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v2drp"]
Nov 01 07:35:45.749481 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.749431    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"]
Nov 01 07:35:45.749931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.749917    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" containerName="revoke-keys"
Nov 01 07:35:45.749981 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.749933    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" containerName="revoke-keys"
Nov 01 07:35:45.750025 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.750015    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" containerName="revoke-keys"
Nov 01 07:35:45.753288 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.753272    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.805022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.804981    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.805212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.805048    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzpc\" (UniqueName: \"kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.805212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.805180    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.807941 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.807915    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:35:45.808103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.808088    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:35:45.808531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.808492    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:35:45.808531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.808497    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:35:45.811227 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.811202    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"]
Nov 01 07:35:45.906505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.906386    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.906505 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.906437    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.906776 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.906605    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzpc\" (UniqueName: \"kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.907149 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.907122    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.909223 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.909192    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:45.964019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:45.963983    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzpc\" (UniqueName: \"kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc\") pod \"maas-api-revoke-sub-3d4156f1-vpc8b\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:46.065431 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:46.065396    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:46.440361 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:46.440332    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"]
Nov 01 07:35:46.442515 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:35:46.442489    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d630f5_cbfa_4f2a_a30c_94edefc1e254.slice/crio-c83d1e92217a9bf5851088d3b5d40b7f6b08d7d9785f709dbf80510072fe726c WatchSource:0}: Error finding container c83d1e92217a9bf5851088d3b5d40b7f6b08d7d9785f709dbf80510072fe726c: Status 404 returned error can't find the container with id c83d1e92217a9bf5851088d3b5d40b7f6b08d7d9785f709dbf80510072fe726c
Nov 01 07:35:46.589974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:46.589938    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5" path="/var/lib/kubelet/pods/c2fe9a1d-cf87-4eb5-89b9-dbdfae9c3ff5/volumes"
Nov 01 07:35:46.784577 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:46.784537    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b" event={"ID":"02d630f5-cbfa-4f2a-a30c-94edefc1e254","Type":"ContainerStarted","Data":"01045aab7a1bf060efb2d3eaf33d5c5e715a721b9b8af853802c046012e707e0"}
Nov 01 07:35:46.784984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:46.784606    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b" event={"ID":"02d630f5-cbfa-4f2a-a30c-94edefc1e254","Type":"ContainerStarted","Data":"c83d1e92217a9bf5851088d3b5d40b7f6b08d7d9785f709dbf80510072fe726c"}
Nov 01 07:35:46.898260 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:46.898204    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b" podStartSLOduration=1.898186462 podStartE2EDuration="1.898186462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:35:46.896707335 +0000 UTC m=+5158.464514166" watchObservedRunningTime="2026-11-01 07:35:46.898186462 +0000 UTC m=+5158.465993292"
Nov 01 07:35:47.792166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:47.792128    4468 generic.go:356] "Generic (PLEG): container finished" podID="02d630f5-cbfa-4f2a-a30c-94edefc1e254" containerID="01045aab7a1bf060efb2d3eaf33d5c5e715a721b9b8af853802c046012e707e0" exitCode=0
Nov 01 07:35:47.792590 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:47.792220    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b" event={"ID":"02d630f5-cbfa-4f2a-a30c-94edefc1e254","Type":"ContainerDied","Data":"01045aab7a1bf060efb2d3eaf33d5c5e715a721b9b8af853802c046012e707e0"}
Nov 01 07:35:48.937582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:48.937555    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:49.038870 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.038831    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca\") pod \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") "
Nov 01 07:35:49.038870 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.038874    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca\") pod \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") "
Nov 01 07:35:49.039092 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.038907    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc\" (UniqueName: \"kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc\") pod \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\" (UID: \"02d630f5-cbfa-4f2a-a30c-94edefc1e254\") "
Nov 01 07:35:49.039280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.039257    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca" pod "02d630f5-cbfa-4f2a-a30c-94edefc1e254" (UID: "02d630f5-cbfa-4f2a-a30c-94edefc1e254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:35:49.041314 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.041283    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca" pod "02d630f5-cbfa-4f2a-a30c-94edefc1e254" (UID: "02d630f5-cbfa-4f2a-a30c-94edefc1e254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:35:49.041314 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.041298    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc" pod "02d630f5-cbfa-4f2a-a30c-94edefc1e254" (UID: "02d630f5-cbfa-4f2a-a30c-94edefc1e254"). InnerVolumeSpecName "kube-api-access-8kzpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:35:49.140588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.140500    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:49.140588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.140532    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d630f5-cbfa-4f2a-a30c-94edefc1e254-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:49.140588 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.140541    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzpc\" (UniqueName: \"kubernetes.io/projected/02d630f5-cbfa-4f2a-a30c-94edefc1e254-kube-api-access-8kzpc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:35:49.803314 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.803286    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"
Nov 01 07:35:49.803519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.803323    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b" event={"ID":"02d630f5-cbfa-4f2a-a30c-94edefc1e254","Type":"ContainerDied","Data":"c83d1e92217a9bf5851088d3b5d40b7f6b08d7d9785f709dbf80510072fe726c"}
Nov 01 07:35:49.803519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:49.803369    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83d1e92217a9bf5851088d3b5d40b7f6b08d7d9785f709dbf80510072fe726c"
Nov 01 07:35:53.157975 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:53.157946    4468 scope.go:122] "RemoveContainer" containerID="7dc75375193b020ad832689f32d8850deb56f062acd24eebabadeae486abce47"
Nov 01 07:35:53.168833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:35:53.168806    4468 scope.go:122] "RemoveContainer" containerID="21b82c6adc719d5b2e1fa0b5559fd82995b6e2229370ca350422ddd2cb955e06"
Nov 01 07:36:02.945885 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:36:02.945807    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:36:11.486529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:11.486495    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:36:20.155613 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.155565    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"]
Nov 01 07:36:20.156166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.155987    4468 cpu_manager.go:414] "containerMap: removing container" podUID="02d630f5-cbfa-4f2a-a30c-94edefc1e254" containerName="revoke-keys"
Nov 01 07:36:20.156166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.155998    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d630f5-cbfa-4f2a-a30c-94edefc1e254" containerName="revoke-keys"
Nov 01 07:36:20.156166 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.156081    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d630f5-cbfa-4f2a-a30c-94edefc1e254" containerName="revoke-keys"
Nov 01 07:36:20.159319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.159299    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.213539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.213459    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:36:20.262915 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.262888    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:36:20.263121 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.263067    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:36:20.263198 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.263180    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:36:20.265556 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.265523    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"]
Nov 01 07:36:20.265935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.265918    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjrr\" (UniqueName: \"kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.265974 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.265952    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.266052 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.266038    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.366997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.366960    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjrr\" (UniqueName: \"kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.367184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.367016    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.367184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.367081    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.367769 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.367750    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.369440 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.369422    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.423665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.423581    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjrr\" (UniqueName: \"kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr\") pod \"maas-api-revoke-sub-e2fc9c5b-m6hdf\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.473482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.473441    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:20.842356 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.842328    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"]
Nov 01 07:36:20.844333 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:36:20.844300    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59499c9d_fbad_4e7e_b5ba_6351a767b110.slice/crio-31fe7ad3c7fdecb3db3d1c80b62bd62a58638ceaf8a8f64d8325e38270cd9c6e WatchSource:0}: Error finding container 31fe7ad3c7fdecb3db3d1c80b62bd62a58638ceaf8a8f64d8325e38270cd9c6e: Status 404 returned error can't find the container with id 31fe7ad3c7fdecb3db3d1c80b62bd62a58638ceaf8a8f64d8325e38270cd9c6e
Nov 01 07:36:20.999972 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:20.999936    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf" event={"ID":"59499c9d-fbad-4e7e-b5ba-6351a767b110","Type":"ContainerStarted","Data":"31fe7ad3c7fdecb3db3d1c80b62bd62a58638ceaf8a8f64d8325e38270cd9c6e"}
Nov 01 07:36:22.007354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:22.007277    4468 generic.go:356] "Generic (PLEG): container finished" podID="59499c9d-fbad-4e7e-b5ba-6351a767b110" containerID="cc15196f9fa4956d457d33bbbabf02f2c435381c844aee6f54c06632ea140687" exitCode=0
Nov 01 07:36:22.007731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:22.007372    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf" event={"ID":"59499c9d-fbad-4e7e-b5ba-6351a767b110","Type":"ContainerDied","Data":"cc15196f9fa4956d457d33bbbabf02f2c435381c844aee6f54c06632ea140687"}
Nov 01 07:36:23.145260 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.145233    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:23.295309 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.295288    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca\") pod \"59499c9d-fbad-4e7e-b5ba-6351a767b110\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") "
Nov 01 07:36:23.295507 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.295397    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca\") pod \"59499c9d-fbad-4e7e-b5ba-6351a767b110\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") "
Nov 01 07:36:23.295507 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.295433    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr\" (UniqueName: \"kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr\") pod \"59499c9d-fbad-4e7e-b5ba-6351a767b110\" (UID: \"59499c9d-fbad-4e7e-b5ba-6351a767b110\") "
Nov 01 07:36:23.295658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.295626    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca" pod "59499c9d-fbad-4e7e-b5ba-6351a767b110" (UID: "59499c9d-fbad-4e7e-b5ba-6351a767b110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:36:23.297665 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.297637    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca" pod "59499c9d-fbad-4e7e-b5ba-6351a767b110" (UID: "59499c9d-fbad-4e7e-b5ba-6351a767b110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:36:23.297783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.297749    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr" pod "59499c9d-fbad-4e7e-b5ba-6351a767b110" (UID: "59499c9d-fbad-4e7e-b5ba-6351a767b110"). InnerVolumeSpecName "kube-api-access-gkjrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:36:23.397157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.397132    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:23.397157 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.397156    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59499c9d-fbad-4e7e-b5ba-6351a767b110-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:23.397354 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:23.397165    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkjrr\" (UniqueName: \"kubernetes.io/projected/59499c9d-fbad-4e7e-b5ba-6351a767b110-kube-api-access-gkjrr\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:24.019319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:24.019288    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"
Nov 01 07:36:24.019319 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:24.019296    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf" event={"ID":"59499c9d-fbad-4e7e-b5ba-6351a767b110","Type":"ContainerDied","Data":"31fe7ad3c7fdecb3db3d1c80b62bd62a58638ceaf8a8f64d8325e38270cd9c6e"}
Nov 01 07:36:24.019554 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:24.019337    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31fe7ad3c7fdecb3db3d1c80b62bd62a58638ceaf8a8f64d8325e38270cd9c6e"
Nov 01 07:36:30.860856 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.860824    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"]
Nov 01 07:36:30.861267 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.861252    4468 cpu_manager.go:414] "containerMap: removing container" podUID="59499c9d-fbad-4e7e-b5ba-6351a767b110" containerName="revoke-keys"
Nov 01 07:36:30.861311 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.861270    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59499c9d-fbad-4e7e-b5ba-6351a767b110" containerName="revoke-keys"
Nov 01 07:36:30.861367 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.861357    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="59499c9d-fbad-4e7e-b5ba-6351a767b110" containerName="revoke-keys"
Nov 01 07:36:30.864762 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.864746    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:30.967766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.967723    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:36:30.967766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.967751    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:36:30.967766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.967760    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:36:30.968021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.967751    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:36:30.968021 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.968006    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:30.968106 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.968048    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:30.968151 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.968118    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxw8n\" (UniqueName: \"kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:30.970353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:30.970334    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"]
Nov 01 07:36:31.069530 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.069506    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.069703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.069547    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxw8n\" (UniqueName: \"kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.069703 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.069631    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.070275 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.070221    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.071969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.071946    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.125607 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.125552    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxw8n\" (UniqueName: \"kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n\") pod \"maas-api-revoke-sub-5a29efb3-zwzml\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.177821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.177802    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:31.551818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:31.551789    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"]
Nov 01 07:36:31.553849 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:36:31.553823    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9b1601_e7ea_4f36_895f_1cae56c8759a.slice/crio-99aa46b6725d05664749b19641e22562c6b9dc1994fa803ddbfc661d250e8dbc WatchSource:0}: Error finding container 99aa46b6725d05664749b19641e22562c6b9dc1994fa803ddbfc661d250e8dbc: Status 404 returned error can't find the container with id 99aa46b6725d05664749b19641e22562c6b9dc1994fa803ddbfc661d250e8dbc
Nov 01 07:36:32.061335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:32.061254    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml" event={"ID":"3f9b1601-e7ea-4f36-895f-1cae56c8759a","Type":"ContainerStarted","Data":"f79f8157f75afd8bdb75fa0b77493f2ff585da6ff18563aa467ec8de698132a8"}
Nov 01 07:36:32.061335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:32.061314    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml" event={"ID":"3f9b1601-e7ea-4f36-895f-1cae56c8759a","Type":"ContainerStarted","Data":"99aa46b6725d05664749b19641e22562c6b9dc1994fa803ddbfc661d250e8dbc"}
Nov 01 07:36:32.175584 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:32.175530    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml" podStartSLOduration=2.175516467 podStartE2EDuration="2.175516467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:36:32.173502292 +0000 UTC m=+5203.741309132" watchObservedRunningTime="2026-11-01 07:36:32.175516467 +0000 UTC m=+5203.743323295"
Nov 01 07:36:32.946265 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:36:32.946232    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:36:32.986289 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:32.986259    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:36:33.068035 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:33.068002    4468 generic.go:356] "Generic (PLEG): container finished" podID="3f9b1601-e7ea-4f36-895f-1cae56c8759a" containerID="f79f8157f75afd8bdb75fa0b77493f2ff585da6ff18563aa467ec8de698132a8" exitCode=0
Nov 01 07:36:33.068387 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:33.068064    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml" event={"ID":"3f9b1601-e7ea-4f36-895f-1cae56c8759a","Type":"ContainerDied","Data":"f79f8157f75afd8bdb75fa0b77493f2ff585da6ff18563aa467ec8de698132a8"}
Nov 01 07:36:34.487850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.487827    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:34.674236 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.673843    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca\") pod \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") "
Nov 01 07:36:34.674430 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.674202    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n\" (UniqueName: \"kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n\") pod \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") "
Nov 01 07:36:34.674883 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.674861    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca\") pod \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\" (UID: \"3f9b1601-e7ea-4f36-895f-1cae56c8759a\") "
Nov 01 07:36:34.682818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.682767    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca" pod "3f9b1601-e7ea-4f36-895f-1cae56c8759a" (UID: "3f9b1601-e7ea-4f36-895f-1cae56c8759a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:36:34.683030 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.683001    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n" pod "3f9b1601-e7ea-4f36-895f-1cae56c8759a" (UID: "3f9b1601-e7ea-4f36-895f-1cae56c8759a"). InnerVolumeSpecName "kube-api-access-jxw8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:36:34.685224 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.685192    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca" pod "3f9b1601-e7ea-4f36-895f-1cae56c8759a" (UID: "3f9b1601-e7ea-4f36-895f-1cae56c8759a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:36:34.776437 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.776387    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:34.776437 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.776428    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxw8n\" (UniqueName: \"kubernetes.io/projected/3f9b1601-e7ea-4f36-895f-1cae56c8759a-kube-api-access-jxw8n\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:34.776437 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:34.776444    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9b1601-e7ea-4f36-895f-1cae56c8759a-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:35.084716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:35.084686    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"
Nov 01 07:36:35.084872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:35.084686    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml" event={"ID":"3f9b1601-e7ea-4f36-895f-1cae56c8759a","Type":"ContainerDied","Data":"99aa46b6725d05664749b19641e22562c6b9dc1994fa803ddbfc661d250e8dbc"}
Nov 01 07:36:35.084872 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:35.084818    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99aa46b6725d05664749b19641e22562c6b9dc1994fa803ddbfc661d250e8dbc"
Nov 01 07:36:43.602660 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:43.602626    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:36:47.627725 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.627691    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"]
Nov 01 07:36:47.677231 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.677203    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vpc8b"]
Nov 01 07:36:47.747716 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.747681    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"]
Nov 01 07:36:47.748155 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.748141    4468 cpu_manager.go:414] "containerMap: removing container" podUID="3f9b1601-e7ea-4f36-895f-1cae56c8759a" containerName="revoke-keys"
Nov 01 07:36:47.748208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.748157    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9b1601-e7ea-4f36-895f-1cae56c8759a" containerName="revoke-keys"
Nov 01 07:36:47.748244 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.748235    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9b1601-e7ea-4f36-895f-1cae56c8759a" containerName="revoke-keys"
Nov 01 07:36:47.751722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.751703    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.801848 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.801825    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.801995 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.801975    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.802057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.802042    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jljq\" (UniqueName: \"kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.805225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.805201    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:36:47.805384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.805277    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:36:47.805384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.805323    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:36:47.805384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.805347    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:36:47.808041 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.808021    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"]
Nov 01 07:36:47.903617 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.903534    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.903784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.903670    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.903784 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.903702    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jljq\" (UniqueName: \"kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.904265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.904244    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.906112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.906092    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:47.960808 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:47.960783    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jljq\" (UniqueName: \"kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq\") pod \"maas-api-revoke-sub-3d4156f1-s7n7r\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:48.064446 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:48.064419    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:48.431570 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:48.431542    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"]
Nov 01 07:36:48.433063 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:36:48.433036    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6416dbc5_9592_4809_bbda_701537faf7a9.slice/crio-a78d43a7890aa92511ed1f844682b3ec13a185dbb94e22a72306379c639c8cfa WatchSource:0}: Error finding container a78d43a7890aa92511ed1f844682b3ec13a185dbb94e22a72306379c639c8cfa: Status 404 returned error can't find the container with id a78d43a7890aa92511ed1f844682b3ec13a185dbb94e22a72306379c639c8cfa
Nov 01 07:36:48.588891 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:48.588861    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d630f5-cbfa-4f2a-a30c-94edefc1e254" path="/var/lib/kubelet/pods/02d630f5-cbfa-4f2a-a30c-94edefc1e254/volumes"
Nov 01 07:36:49.160105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:49.160070    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r" event={"ID":"6416dbc5-9592-4809-bbda-701537faf7a9","Type":"ContainerStarted","Data":"c81e9454409fb92a4580f42cbb8c17ac887e7292b77af3c431dbd717bd836996"}
Nov 01 07:36:49.160538 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:49.160122    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r" event={"ID":"6416dbc5-9592-4809-bbda-701537faf7a9","Type":"ContainerStarted","Data":"a78d43a7890aa92511ed1f844682b3ec13a185dbb94e22a72306379c639c8cfa"}
Nov 01 07:36:49.272781 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:49.272737    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r" podStartSLOduration=2.27272494 podStartE2EDuration="2.27272494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:36:49.271199496 +0000 UTC m=+5220.839006322" watchObservedRunningTime="2026-11-01 07:36:49.27272494 +0000 UTC m=+5220.840531781"
Nov 01 07:36:50.167220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:50.167188    4468 generic.go:356] "Generic (PLEG): container finished" podID="6416dbc5-9592-4809-bbda-701537faf7a9" containerID="c81e9454409fb92a4580f42cbb8c17ac887e7292b77af3c431dbd717bd836996" exitCode=0
Nov 01 07:36:50.167643 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:50.167230    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r" event={"ID":"6416dbc5-9592-4809-bbda-701537faf7a9","Type":"ContainerDied","Data":"c81e9454409fb92a4580f42cbb8c17ac887e7292b77af3c431dbd717bd836996"}
Nov 01 07:36:51.297173 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.297150    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:51.435997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.435923    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca\") pod \"6416dbc5-9592-4809-bbda-701537faf7a9\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") "
Nov 01 07:36:51.435997 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.435965    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq\" (UniqueName: \"kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq\") pod \"6416dbc5-9592-4809-bbda-701537faf7a9\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") "
Nov 01 07:36:51.436185 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.436041    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca\") pod \"6416dbc5-9592-4809-bbda-701537faf7a9\" (UID: \"6416dbc5-9592-4809-bbda-701537faf7a9\") "
Nov 01 07:36:51.436443 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.436417    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca" pod "6416dbc5-9592-4809-bbda-701537faf7a9" (UID: "6416dbc5-9592-4809-bbda-701537faf7a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:36:51.438047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.438021    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca" pod "6416dbc5-9592-4809-bbda-701537faf7a9" (UID: "6416dbc5-9592-4809-bbda-701537faf7a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:36:51.438187 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.438163    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq" pod "6416dbc5-9592-4809-bbda-701537faf7a9" (UID: "6416dbc5-9592-4809-bbda-701537faf7a9"). InnerVolumeSpecName "kube-api-access-4jljq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:36:51.537107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.537080    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:51.537107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.537108    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6416dbc5-9592-4809-bbda-701537faf7a9-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:51.537285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:51.537118    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jljq\" (UniqueName: \"kubernetes.io/projected/6416dbc5-9592-4809-bbda-701537faf7a9-kube-api-access-4jljq\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:36:52.178600 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:52.178563    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"
Nov 01 07:36:52.178600 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:52.178589    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r" event={"ID":"6416dbc5-9592-4809-bbda-701537faf7a9","Type":"ContainerDied","Data":"a78d43a7890aa92511ed1f844682b3ec13a185dbb94e22a72306379c639c8cfa"}
Nov 01 07:36:52.178817 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:36:52.178644    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78d43a7890aa92511ed1f844682b3ec13a185dbb94e22a72306379c639c8cfa"
Nov 01 07:37:02.951362 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:37:02.951321    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:37:19.043235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:19.043204    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"]
Nov 01 07:37:19.078140 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:19.078111    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bllb6"]
Nov 01 07:37:20.588717 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:20.588675    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5106f047-8c43-4d70-a733-30c42d4abc7d" path="/var/lib/kubelet/pods/5106f047-8c43-4d70-a733-30c42d4abc7d/volumes"
Nov 01 07:37:22.805146 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:22.805108    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:37:32.946518 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:37:32.946409    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:37:41.698091 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:41.698056    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:37:49.044534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:49.044493    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"]
Nov 01 07:37:49.093931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:49.093895    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6rn8g"]
Nov 01 07:37:50.590835 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:37:50.590791    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0" path="/var/lib/kubelet/pods/6cdb334b-3ba9-4201-91d1-c8ccea8fd1b0/volumes"
Nov 01 07:38:02.043717 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:02.043682    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"]
Nov 01 07:38:02.093222 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:02.093190    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8h9g6"]
Nov 01 07:38:02.589849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:02.589809    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a25fdc-38cf-418e-9574-bdb204bb0d6b" path="/var/lib/kubelet/pods/d5a25fdc-38cf-418e-9574-bdb204bb0d6b/volumes"
Nov 01 07:38:02.946970 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:38:02.946880    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:38:07.489195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:07.489155    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:38:22.062234 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.062194    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"]
Nov 01 07:38:22.062888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.062854    4468 cpu_manager.go:414] "containerMap: removing container" podUID="6416dbc5-9592-4809-bbda-701537faf7a9" containerName="revoke-keys"
Nov 01 07:38:22.062888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.062873    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6416dbc5-9592-4809-bbda-701537faf7a9" containerName="revoke-keys"
Nov 01 07:38:22.063018 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.063001    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="6416dbc5-9592-4809-bbda-701537faf7a9" containerName="revoke-keys"
Nov 01 07:38:22.066161 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.066141    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.120270 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.120228    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:38:22.120436 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.120288    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:38:22.169447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.169418    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:38:22.169653 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.169559    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:38:22.171892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.171867    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"]
Nov 01 07:38:22.228705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.228674    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.228873 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.228850    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfp7\" (UniqueName: \"kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.228936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.228894    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.330285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.330217    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.330402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.330290    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfp7\" (UniqueName: \"kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.330402 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.330313    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.330849 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.330823    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.332682 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.332666    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.386536 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.386499    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfp7\" (UniqueName: \"kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7\") pod \"maas-api-revoke-sub-baece1ba-rpntn\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:22.678615 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:22.678544    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:23.047112 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:23.047087    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"]
Nov 01 07:38:23.048769 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:38:23.048742    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e78fe45_8621_4aac_be12_3006a055454f.slice/crio-cdc91a05adfd31aca11b9cfae4d11b8650e83701c53147827f4958b7b34e7c71 WatchSource:0}: Error finding container cdc91a05adfd31aca11b9cfae4d11b8650e83701c53147827f4958b7b34e7c71: Status 404 returned error can't find the container with id cdc91a05adfd31aca11b9cfae4d11b8650e83701c53147827f4958b7b34e7c71
Nov 01 07:38:23.654673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:23.654589    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn" event={"ID":"2e78fe45-8621-4aac-be12-3006a055454f","Type":"ContainerStarted","Data":"b252bb00720a858d535b46905dfc9b7863fd632c8871c025a05a565908f655af"}
Nov 01 07:38:23.654673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:23.654646    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn" event={"ID":"2e78fe45-8621-4aac-be12-3006a055454f","Type":"ContainerStarted","Data":"cdc91a05adfd31aca11b9cfae4d11b8650e83701c53147827f4958b7b34e7c71"}
Nov 01 07:38:23.766992 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:23.766679    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn" podStartSLOduration=1.766665246 podStartE2EDuration="1.766665246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:38:23.765656207 +0000 UTC m=+5315.333463037" watchObservedRunningTime="2026-11-01 07:38:23.766665246 +0000 UTC m=+5315.334472076"
Nov 01 07:38:24.661547 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:24.661509    4468 generic.go:356] "Generic (PLEG): container finished" podID="2e78fe45-8621-4aac-be12-3006a055454f" containerID="b252bb00720a858d535b46905dfc9b7863fd632c8871c025a05a565908f655af" exitCode=0
Nov 01 07:38:24.661927 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:24.661592    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn" event={"ID":"2e78fe45-8621-4aac-be12-3006a055454f","Type":"ContainerDied","Data":"b252bb00720a858d535b46905dfc9b7863fd632c8871c025a05a565908f655af"}
Nov 01 07:38:25.797105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.797086    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:25.862245 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.862219    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7\" (UniqueName: \"kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7\") pod \"2e78fe45-8621-4aac-be12-3006a055454f\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") "
Nov 01 07:38:25.862384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.862263    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca\") pod \"2e78fe45-8621-4aac-be12-3006a055454f\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") "
Nov 01 07:38:25.862384 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.862288    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca\") pod \"2e78fe45-8621-4aac-be12-3006a055454f\" (UID: \"2e78fe45-8621-4aac-be12-3006a055454f\") "
Nov 01 07:38:25.862649 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.862624    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca" pod "2e78fe45-8621-4aac-be12-3006a055454f" (UID: "2e78fe45-8621-4aac-be12-3006a055454f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:38:25.864403 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.864383    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca" pod "2e78fe45-8621-4aac-be12-3006a055454f" (UID: "2e78fe45-8621-4aac-be12-3006a055454f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:38:25.864548 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.864531    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7" pod "2e78fe45-8621-4aac-be12-3006a055454f" (UID: "2e78fe45-8621-4aac-be12-3006a055454f"). InnerVolumeSpecName "kube-api-access-8qfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:38:25.964139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.964109    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qfp7\" (UniqueName: \"kubernetes.io/projected/2e78fe45-8621-4aac-be12-3006a055454f-kube-api-access-8qfp7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:38:25.964139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.964137    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:38:25.964139 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:25.964147    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e78fe45-8621-4aac-be12-3006a055454f-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:38:26.674110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:26.674084    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"
Nov 01 07:38:26.674110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:26.674094    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn" event={"ID":"2e78fe45-8621-4aac-be12-3006a055454f","Type":"ContainerDied","Data":"cdc91a05adfd31aca11b9cfae4d11b8650e83701c53147827f4958b7b34e7c71"}
Nov 01 07:38:26.674298 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:26.674136    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc91a05adfd31aca11b9cfae4d11b8650e83701c53147827f4958b7b34e7c71"
Nov 01 07:38:32.946257 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:38:32.946221    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:38:36.806418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:36.806382    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:38:37.044729 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:37.044689    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"]
Nov 01 07:38:37.094005 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:37.093942    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wdzpp"]
Nov 01 07:38:38.588618 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:38.588583    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756d76d9-67b9-4f9e-b3eb-2b39752357f1" path="/var/lib/kubelet/pods/756d76d9-67b9-4f9e-b3eb-2b39752357f1/volumes"
Nov 01 07:38:45.488885 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:45.488851    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:38:48.043940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:48.043903    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"]
Nov 01 07:38:48.093051 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:48.093017    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-86n6z"]
Nov 01 07:38:48.589666 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:48.589631    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334fade6-bcfa-4fe1-ad1d-318efa3c63eb" path="/var/lib/kubelet/pods/334fade6-bcfa-4fe1-ad1d-318efa3c63eb/volumes"
Nov 01 07:38:53.210684 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:53.210652    4468 scope.go:122] "RemoveContainer" containerID="f144ca51c4a790831c4fc4f6654778a94ebc223335ef3b0c73de97ea693b7f44"
Nov 01 07:38:53.219448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:38:53.219430    4468 scope.go:122] "RemoveContainer" containerID="d462a3578b1506f4645c067d5cd289046bfc8a74be88212968ef33d370717a89"
Nov 01 07:39:02.946785 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:39:02.946702    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:39:21.082531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:21.082487    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:39:23.044200 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:23.044166    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"]
Nov 01 07:39:23.093905 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:23.093872    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-49jjf"]
Nov 01 07:39:24.589086 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:24.589050    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80da0a69-7c04-4e4f-b07f-aafa7e11bf7f" path="/var/lib/kubelet/pods/80da0a69-7c04-4e4f-b07f-aafa7e11bf7f/volumes"
Nov 01 07:39:27.946493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:27.946430    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"]
Nov 01 07:39:27.947111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:27.947090    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2e78fe45-8621-4aac-be12-3006a055454f" containerName="revoke-keys"
Nov 01 07:39:27.947204 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:27.947113    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e78fe45-8621-4aac-be12-3006a055454f" containerName="revoke-keys"
Nov 01 07:39:27.947261 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:27.947211    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e78fe45-8621-4aac-be12-3006a055454f" containerName="revoke-keys"
Nov 01 07:39:27.950661 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:27.950640    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.039378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.039345    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjqf\" (UniqueName: \"kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.039554 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.039389    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.039614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.039563    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.054438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.054411    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:39:28.054602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.054511    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:39:28.054602 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.054558    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:39:28.054714 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.054644    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:39:28.056752 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.056736    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"]
Nov 01 07:39:28.140903 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.140872    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjqf\" (UniqueName: \"kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.141100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.140913    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.141100 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.141043    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.141756 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.141736    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.143383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.143364    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.197341 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.197270    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjqf\" (UniqueName: \"kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.263627 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.263599    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:28.631671 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.631647    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"]
Nov 01 07:39:28.633352 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:39:28.633326    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3071fc_9ac6_46b8_a1be_e15e20b3acda.slice/crio-8f42e437f28773b47a02eb1fa71b8bf5c81cfc0c76f4e3d0ca9062d38c93a568 WatchSource:0}: Error finding container 8f42e437f28773b47a02eb1fa71b8bf5c81cfc0c76f4e3d0ca9062d38c93a568: Status 404 returned error can't find the container with id 8f42e437f28773b47a02eb1fa71b8bf5c81cfc0c76f4e3d0ca9062d38c93a568
Nov 01 07:39:28.996383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.996342    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh" event={"ID":"0e3071fc-9ac6-46b8-a1be-e15e20b3acda","Type":"ContainerStarted","Data":"979ac5e17d289f302a484d579cca50e00047f4068a9798282a1035fe60e5d81b"}
Nov 01 07:39:28.996839 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:28.996389    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh" event={"ID":"0e3071fc-9ac6-46b8-a1be-e15e20b3acda","Type":"ContainerStarted","Data":"8f42e437f28773b47a02eb1fa71b8bf5c81cfc0c76f4e3d0ca9062d38c93a568"}
Nov 01 07:39:29.111056 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:29.110999    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh" podStartSLOduration=2.110985554 podStartE2EDuration="2.110985554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:39:29.109122229 +0000 UTC m=+5380.676929057" watchObservedRunningTime="2026-11-01 07:39:29.110985554 +0000 UTC m=+5380.678792395"
Nov 01 07:39:30.004818 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:30.004783    4468 generic.go:356] "Generic (PLEG): container finished" podID="0e3071fc-9ac6-46b8-a1be-e15e20b3acda" containerID="979ac5e17d289f302a484d579cca50e00047f4068a9798282a1035fe60e5d81b" exitCode=0
Nov 01 07:39:30.005235 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:30.004864    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh" event={"ID":"0e3071fc-9ac6-46b8-a1be-e15e20b3acda","Type":"ContainerDied","Data":"979ac5e17d289f302a484d579cca50e00047f4068a9798282a1035fe60e5d81b"}
Nov 01 07:39:31.143705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.143684    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:31.275116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.275026    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca\") pod \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") "
Nov 01 07:39:31.275266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.275173    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf\" (UniqueName: \"kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf\") pod \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") "
Nov 01 07:39:31.275266 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.275246    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca\") pod \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\" (UID: \"0e3071fc-9ac6-46b8-a1be-e15e20b3acda\") "
Nov 01 07:39:31.275628 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.275599    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca" pod "0e3071fc-9ac6-46b8-a1be-e15e20b3acda" (UID: "0e3071fc-9ac6-46b8-a1be-e15e20b3acda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:39:31.277368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.277346    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf" pod "0e3071fc-9ac6-46b8-a1be-e15e20b3acda" (UID: "0e3071fc-9ac6-46b8-a1be-e15e20b3acda"). InnerVolumeSpecName "kube-api-access-wtjqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:39:31.277529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.277345    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca" pod "0e3071fc-9ac6-46b8-a1be-e15e20b3acda" (UID: "0e3071fc-9ac6-46b8-a1be-e15e20b3acda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:39:31.376592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.376562    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjqf\" (UniqueName: \"kubernetes.io/projected/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-kube-api-access-wtjqf\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:39:31.376592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.376587    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:39:31.376592 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:31.376597    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3071fc-9ac6-46b8-a1be-e15e20b3acda-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:39:32.015860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:32.015827    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"
Nov 01 07:39:32.016058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:32.015869    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh" event={"ID":"0e3071fc-9ac6-46b8-a1be-e15e20b3acda","Type":"ContainerDied","Data":"8f42e437f28773b47a02eb1fa71b8bf5c81cfc0c76f4e3d0ca9062d38c93a568"}
Nov 01 07:39:32.016058 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:32.015922    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f42e437f28773b47a02eb1fa71b8bf5c81cfc0c76f4e3d0ca9062d38c93a568"
Nov 01 07:39:32.946522 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:39:32.946491    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:39:42.044008 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:42.043969    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"]
Nov 01 07:39:42.093543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:42.093513    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qbjx5"]
Nov 01 07:39:42.588770 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:42.588736    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311da33e-8a35-4489-9054-b99c200c22cc" path="/var/lib/kubelet/pods/311da33e-8a35-4489-9054-b99c200c22cc/volumes"
Nov 01 07:39:45.666669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:45.666630    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"]
Nov 01 07:39:45.716470 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:45.716445    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2624b0-mlwbh"]
Nov 01 07:39:46.589658 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:46.589621    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3071fc-9ac6-46b8-a1be-e15e20b3acda" path="/var/lib/kubelet/pods/0e3071fc-9ac6-46b8-a1be-e15e20b3acda/volumes"
Nov 01 07:39:47.585407 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.585376    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"]
Nov 01 07:39:47.634773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.634741    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7n7r"]
Nov 01 07:39:47.757529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.757496    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"]
Nov 01 07:39:47.757890 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.757878    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0e3071fc-9ac6-46b8-a1be-e15e20b3acda" containerName="revoke-keys"
Nov 01 07:39:47.757935 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.757891    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3071fc-9ac6-46b8-a1be-e15e20b3acda" containerName="revoke-keys"
Nov 01 07:39:47.757987 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.757979    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3071fc-9ac6-46b8-a1be-e15e20b3acda" containerName="revoke-keys"
Nov 01 07:39:47.761169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.761154    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:47.815257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.815231    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:39:47.815257 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.815251    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:39:47.815496 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.815415    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:39:47.815603 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.815584    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:39:47.817633 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.817607    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"]
Nov 01 07:39:47.931225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.931151    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:47.931225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.931184    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:47.931225 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:47.931211    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26r7\" (UniqueName: \"kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.032105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.032074    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.032303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.032108    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.032303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.032133    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26r7\" (UniqueName: \"kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.032779 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.032755    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.034479 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.034450    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.088209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.088182    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26r7\" (UniqueName: \"kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7\") pod \"maas-api-revoke-sub-3d4156f1-pwkpj\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.373589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.373564    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:48.589315 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.589281    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6416dbc5-9592-4809-bbda-701537faf7a9" path="/var/lib/kubelet/pods/6416dbc5-9592-4809-bbda-701537faf7a9/volumes"
Nov 01 07:39:48.741749 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:48.741727    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"]
Nov 01 07:39:48.744039 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:39:48.744005    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c60c3f_90ca_4c68_89a0_3d63914ca408.slice/crio-67a9c807bb876096240c5d753d2aa77d35e97d25bb8cb540a26a6cf09e7e34a7 WatchSource:0}: Error finding container 67a9c807bb876096240c5d753d2aa77d35e97d25bb8cb540a26a6cf09e7e34a7: Status 404 returned error can't find the container with id 67a9c807bb876096240c5d753d2aa77d35e97d25bb8cb540a26a6cf09e7e34a7
Nov 01 07:39:49.103985 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:49.103953    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj" event={"ID":"34c60c3f-90ca-4c68-89a0-3d63914ca408","Type":"ContainerStarted","Data":"25b3821c3bf912dc150de888ccfe4ca12c0cfa603bafe35dc88f14474cd3057d"}
Nov 01 07:39:49.104130 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:49.104004    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj" event={"ID":"34c60c3f-90ca-4c68-89a0-3d63914ca408","Type":"ContainerStarted","Data":"67a9c807bb876096240c5d753d2aa77d35e97d25bb8cb540a26a6cf09e7e34a7"}
Nov 01 07:39:49.217452 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:49.217402    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj" podStartSLOduration=2.217386409 podStartE2EDuration="2.217386409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:39:49.216073988 +0000 UTC m=+5400.783880817" watchObservedRunningTime="2026-11-01 07:39:49.217386409 +0000 UTC m=+5400.785193237"
Nov 01 07:39:49.708285 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:49.708251    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:39:50.110934 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:50.110903    4468 generic.go:356] "Generic (PLEG): container finished" podID="34c60c3f-90ca-4c68-89a0-3d63914ca408" containerID="25b3821c3bf912dc150de888ccfe4ca12c0cfa603bafe35dc88f14474cd3057d" exitCode=0
Nov 01 07:39:50.111097 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:50.110984    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj" event={"ID":"34c60c3f-90ca-4c68-89a0-3d63914ca408","Type":"ContainerDied","Data":"25b3821c3bf912dc150de888ccfe4ca12c0cfa603bafe35dc88f14474cd3057d"}
Nov 01 07:39:51.246224 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.246194    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:51.369614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.369535    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca\") pod \"34c60c3f-90ca-4c68-89a0-3d63914ca408\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") "
Nov 01 07:39:51.369614 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.369590    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7\" (UniqueName: \"kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7\") pod \"34c60c3f-90ca-4c68-89a0-3d63914ca408\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") "
Nov 01 07:39:51.369844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.369678    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca\") pod \"34c60c3f-90ca-4c68-89a0-3d63914ca408\" (UID: \"34c60c3f-90ca-4c68-89a0-3d63914ca408\") "
Nov 01 07:39:51.369943 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.369915    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca" pod "34c60c3f-90ca-4c68-89a0-3d63914ca408" (UID: "34c60c3f-90ca-4c68-89a0-3d63914ca408"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:39:51.370057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.370039    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:39:51.371791 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.371771    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7" pod "34c60c3f-90ca-4c68-89a0-3d63914ca408" (UID: "34c60c3f-90ca-4c68-89a0-3d63914ca408"). InnerVolumeSpecName "kube-api-access-r26r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:39:51.371889 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.371856    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca" pod "34c60c3f-90ca-4c68-89a0-3d63914ca408" (UID: "34c60c3f-90ca-4c68-89a0-3d63914ca408"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:39:51.471522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.471492    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c60c3f-90ca-4c68-89a0-3d63914ca408-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:39:51.471522 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:51.471517    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26r7\" (UniqueName: \"kubernetes.io/projected/34c60c3f-90ca-4c68-89a0-3d63914ca408-kube-api-access-r26r7\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:39:52.122649 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:52.122617    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"
Nov 01 07:39:52.122847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:52.122644    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj" event={"ID":"34c60c3f-90ca-4c68-89a0-3d63914ca408","Type":"ContainerDied","Data":"67a9c807bb876096240c5d753d2aa77d35e97d25bb8cb540a26a6cf09e7e34a7"}
Nov 01 07:39:52.122847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:52.122689    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a9c807bb876096240c5d753d2aa77d35e97d25bb8cb540a26a6cf09e7e34a7"
Nov 01 07:39:53.246208 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:53.246173    4468 scope.go:122] "RemoveContainer" containerID="877f2d0bff30f3129bd59053db02144a16ee69d9c12dbce84d301d2556f5287c"
Nov 01 07:39:53.254816 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:53.254790    4468 scope.go:122] "RemoveContainer" containerID="a6d96c4cdc7645435d9fadb35b44d0b00df789cc788ef6e684778f1bb6fbf229"
Nov 01 07:39:53.262589 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:39:53.262572    4468 scope.go:122] "RemoveContainer" containerID="27fbce60971da4192961487cc218e384a6b0026b33d3ae6f7e5d37bc076aa33b"
Nov 01 07:40:02.947145 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:40:02.947112    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:40:03.044093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:03.044059    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"]
Nov 01 07:40:03.046142 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:03.046123    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cx26c"]
Nov 01 07:40:04.588601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:04.588559    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a717f1e7-47e0-4429-ae32-cc71af3044dc" path="/var/lib/kubelet/pods/a717f1e7-47e0-4429-ae32-cc71af3044dc/volumes"
Nov 01 07:40:07.192044 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:07.192011    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:40:11.549213 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.549176    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"]
Nov 01 07:40:11.549660 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.549650    4468 cpu_manager.go:414] "containerMap: removing container" podUID="34c60c3f-90ca-4c68-89a0-3d63914ca408" containerName="revoke-keys"
Nov 01 07:40:11.549708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.549664    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c60c3f-90ca-4c68-89a0-3d63914ca408" containerName="revoke-keys"
Nov 01 07:40:11.549741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.549731    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c60c3f-90ca-4c68-89a0-3d63914ca408" containerName="revoke-keys"
Nov 01 07:40:11.552955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.552940    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.609904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.609864    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"]
Nov 01 07:40:11.656663 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.656628    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:40:11.656663 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.656642    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:40:11.656848 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.656656    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:40:11.656848 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.656730    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:40:11.696979 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.696955    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.697095 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.696987    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.697095 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.697020    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6stc\" (UniqueName: \"kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.797753 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.797724    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.797877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.797762    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.797877 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.797793    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6stc\" (UniqueName: \"kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.798365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.798348    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.800169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.800121    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.854557 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.854525    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6stc\" (UniqueName: \"kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc\") pod \"maas-api-revoke-sub-d2bc93bc-p9l8r\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:11.865519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:11.865499    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:12.231970 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:12.231937    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"]
Nov 01 07:40:12.233735 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:40:12.233692    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6baa1a7_5464_40e9_932a_6c1683c4f252.slice/crio-92b7f7836c42d8294d8f1f1004721b38d541432c84f67aa7c0c692031aa6cdb8 WatchSource:0}: Error finding container 92b7f7836c42d8294d8f1f1004721b38d541432c84f67aa7c0c692031aa6cdb8: Status 404 returned error can't find the container with id 92b7f7836c42d8294d8f1f1004721b38d541432c84f67aa7c0c692031aa6cdb8
Nov 01 07:40:12.245850 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:12.245820    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r" event={"ID":"f6baa1a7-5464-40e9-932a-6c1683c4f252","Type":"ContainerStarted","Data":"92b7f7836c42d8294d8f1f1004721b38d541432c84f67aa7c0c692031aa6cdb8"}
Nov 01 07:40:13.252582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:13.252551    4468 generic.go:356] "Generic (PLEG): container finished" podID="f6baa1a7-5464-40e9-932a-6c1683c4f252" containerID="9b0182475d7b3c5706944886c16abbdf00165e371c59a225a6931b84a8a1d153" exitCode=0
Nov 01 07:40:13.252937 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:13.252633    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r" event={"ID":"f6baa1a7-5464-40e9-932a-6c1683c4f252","Type":"ContainerDied","Data":"9b0182475d7b3c5706944886c16abbdf00165e371c59a225a6931b84a8a1d153"}
Nov 01 07:40:14.382061 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.382039    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:14.523175 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.523104    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc\" (UniqueName: \"kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc\") pod \"f6baa1a7-5464-40e9-932a-6c1683c4f252\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") "
Nov 01 07:40:14.523333 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.523223    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca\") pod \"f6baa1a7-5464-40e9-932a-6c1683c4f252\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") "
Nov 01 07:40:14.523333 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.523270    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca\") pod \"f6baa1a7-5464-40e9-932a-6c1683c4f252\" (UID: \"f6baa1a7-5464-40e9-932a-6c1683c4f252\") "
Nov 01 07:40:14.523686 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.523651    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca" pod "f6baa1a7-5464-40e9-932a-6c1683c4f252" (UID: "f6baa1a7-5464-40e9-932a-6c1683c4f252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:40:14.525380 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.525358    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca" pod "f6baa1a7-5464-40e9-932a-6c1683c4f252" (UID: "f6baa1a7-5464-40e9-932a-6c1683c4f252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:40:14.525506 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.525416    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc" pod "f6baa1a7-5464-40e9-932a-6c1683c4f252" (UID: "f6baa1a7-5464-40e9-932a-6c1683c4f252"). InnerVolumeSpecName "kube-api-access-v6stc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:40:14.624276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.624251    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:40:14.624276 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.624276    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6baa1a7-5464-40e9-932a-6c1683c4f252-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:40:14.624428 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:14.624285    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6stc\" (UniqueName: \"kubernetes.io/projected/f6baa1a7-5464-40e9-932a-6c1683c4f252-kube-api-access-v6stc\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:40:15.264860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:15.264832    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"
Nov 01 07:40:15.264860 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:15.264851    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r" event={"ID":"f6baa1a7-5464-40e9-932a-6c1683c4f252","Type":"ContainerDied","Data":"92b7f7836c42d8294d8f1f1004721b38d541432c84f67aa7c0c692031aa6cdb8"}
Nov 01 07:40:15.265070 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:15.264893    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b7f7836c42d8294d8f1f1004721b38d541432c84f67aa7c0c692031aa6cdb8"
Nov 01 07:40:16.212795 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:16.212767    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:40:20.043662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:20.043633    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"]
Nov 01 07:40:20.093519 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:20.093492    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q8qfp"]
Nov 01 07:40:20.588019 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:20.587990    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f7ab1e-4764-4e87-a906-685cd6e02265" path="/var/lib/kubelet/pods/38f7ab1e-4764-4e87-a906-685cd6e02265/volumes"
Nov 01 07:40:32.949606 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:40:32.949510    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:40:37.083844 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:37.083814    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:40:48.639621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.639589    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"]
Nov 01 07:40:48.639990 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.639963    4468 cpu_manager.go:414] "containerMap: removing container" podUID="f6baa1a7-5464-40e9-932a-6c1683c4f252" containerName="revoke-keys"
Nov 01 07:40:48.639990 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.639976    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6baa1a7-5464-40e9-932a-6c1683c4f252" containerName="revoke-keys"
Nov 01 07:40:48.640071 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.640053    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6baa1a7-5464-40e9-932a-6c1683c4f252" containerName="revoke-keys"
Nov 01 07:40:48.643172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.643156    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.697089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.697062    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:40:48.699983 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.699964    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"]
Nov 01 07:40:48.713604 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.713580    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.713731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.713656    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ffl\" (UniqueName: \"kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.713830 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.713813    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.746874 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.746849    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:40:48.746969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.746895    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:40:48.746969 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.746927    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:40:48.814691 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.814670    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ffl\" (UniqueName: \"kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.814783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.814735    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.814783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.814769    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.815415 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.815398    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.817104 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.817086    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:48.870916 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:48.870889    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ffl\" (UniqueName: \"kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl\") pod \"maas-api-revoke-sub-5f02c688-dgj9q\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:49.006765 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:49.006714    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:40:49.016110 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:49.016082    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:49.381510 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:49.381486    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"]
Nov 01 07:40:49.383576 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:40:49.383544    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0fc632_70fa_41bb_9659_a377652bff32.slice/crio-79d41f6c258d3a1651193cdfa3d9a5ca1844bb92d3cfb0b3a3c96a6d0496da39 WatchSource:0}: Error finding container 79d41f6c258d3a1651193cdfa3d9a5ca1844bb92d3cfb0b3a3c96a6d0496da39: Status 404 returned error can't find the container with id 79d41f6c258d3a1651193cdfa3d9a5ca1844bb92d3cfb0b3a3c96a6d0496da39
Nov 01 07:40:49.465343 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:49.465317    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q" event={"ID":"5d0fc632-70fa-41bb-9659-a377652bff32","Type":"ContainerStarted","Data":"79d41f6c258d3a1651193cdfa3d9a5ca1844bb92d3cfb0b3a3c96a6d0496da39"}
Nov 01 07:40:50.472421 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:50.472387    4468 generic.go:356] "Generic (PLEG): container finished" podID="5d0fc632-70fa-41bb-9659-a377652bff32" containerID="66649cd00f6153453f9e9877516633f140bcad878ad23195e24cfb3cee25d6f3" exitCode=0
Nov 01 07:40:50.472822 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:50.472496    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q" event={"ID":"5d0fc632-70fa-41bb-9659-a377652bff32","Type":"ContainerDied","Data":"66649cd00f6153453f9e9877516633f140bcad878ad23195e24cfb3cee25d6f3"}
Nov 01 07:40:51.608667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.608641    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:51.633919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.633894    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca\") pod \"5d0fc632-70fa-41bb-9659-a377652bff32\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") "
Nov 01 07:40:51.634024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.633932    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl\" (UniqueName: \"kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl\") pod \"5d0fc632-70fa-41bb-9659-a377652bff32\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") "
Nov 01 07:40:51.634024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.633965    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca\") pod \"5d0fc632-70fa-41bb-9659-a377652bff32\" (UID: \"5d0fc632-70fa-41bb-9659-a377652bff32\") "
Nov 01 07:40:51.634295 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.634266    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca" pod "5d0fc632-70fa-41bb-9659-a377652bff32" (UID: "5d0fc632-70fa-41bb-9659-a377652bff32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:40:51.636022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.636000    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca" pod "5d0fc632-70fa-41bb-9659-a377652bff32" (UID: "5d0fc632-70fa-41bb-9659-a377652bff32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:40:51.636022 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.636000    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl" pod "5d0fc632-70fa-41bb-9659-a377652bff32" (UID: "5d0fc632-70fa-41bb-9659-a377652bff32"). InnerVolumeSpecName "kube-api-access-49ffl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:40:51.735951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.735887    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:40:51.735951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.735920    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fc632-70fa-41bb-9659-a377652bff32-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:40:51.735951 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:51.735935    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ffl\" (UniqueName: \"kubernetes.io/projected/5d0fc632-70fa-41bb-9659-a377652bff32-kube-api-access-49ffl\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:40:52.486702 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:52.486449    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q"
Nov 01 07:40:52.486702 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:52.486537    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f02c688-dgj9q" event={"ID":"5d0fc632-70fa-41bb-9659-a377652bff32","Type":"ContainerDied","Data":"79d41f6c258d3a1651193cdfa3d9a5ca1844bb92d3cfb0b3a3c96a6d0496da39"}
Nov 01 07:40:52.486702 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:52.486622    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d41f6c258d3a1651193cdfa3d9a5ca1844bb92d3cfb0b3a3c96a6d0496da39"
Nov 01 07:40:53.290724 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:53.290694    4468 scope.go:122] "RemoveContainer" containerID="e63b4d347aaf99dfe00350dd6de907a2f6ac4172f3212530ba990e98e1d28a4a"
Nov 01 07:40:53.299553 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:53.299529    4468 scope.go:122] "RemoveContainer" containerID="ca4da3abb618ce205c706096784ece775e217a0fb7b51e588e406bb98738ef13"
Nov 01 07:40:59.433583 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.433547    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"]
Nov 01 07:40:59.434093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.433978    4468 cpu_manager.go:414] "containerMap: removing container" podUID="5d0fc632-70fa-41bb-9659-a377652bff32" containerName="revoke-keys"
Nov 01 07:40:59.434093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.433989    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0fc632-70fa-41bb-9659-a377652bff32" containerName="revoke-keys"
Nov 01 07:40:59.434093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.434050    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0fc632-70fa-41bb-9659-a377652bff32" containerName="revoke-keys"
Nov 01 07:40:59.438675 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.438651    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.495357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.495323    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"]
Nov 01 07:40:59.508162 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.508134    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs628\" (UniqueName: \"kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.508291 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.508164    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.508335 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.508290    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.541350 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.541318    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:40:59.541540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.541400    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:40:59.541540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.541460    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:40:59.541540 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.541534    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:40:59.609171 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.609135    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs628\" (UniqueName: \"kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.609337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.609183    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.609337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.609217    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.609980 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.609961    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.611449 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.611428    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.665457 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.665435    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs628\" (UniqueName: \"kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628\") pod \"maas-api-revoke-sub-47c764ac-xphqg\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:40:59.754204 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:40:59.754132    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:41:00.121106 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:00.121073    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"]
Nov 01 07:41:00.122765 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:41:00.122739    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fb1ef2_fc7b_4e5f_8728_2dbb762ddbf0.slice/crio-2c093b1619555ca6dbb2b91ccb7e3ada144c7a0e91c3b4dfd618bdb5a009f409 WatchSource:0}: Error finding container 2c093b1619555ca6dbb2b91ccb7e3ada144c7a0e91c3b4dfd618bdb5a009f409: Status 404 returned error can't find the container with id 2c093b1619555ca6dbb2b91ccb7e3ada144c7a0e91c3b4dfd618bdb5a009f409
Nov 01 07:41:00.527833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:00.527749    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg" event={"ID":"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0","Type":"ContainerStarted","Data":"64dd20bd01a93d7f29c03f87d0eccc2cff638666e213a01e4683476a86d598ef"}
Nov 01 07:41:00.527833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:00.527810    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg" event={"ID":"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0","Type":"ContainerStarted","Data":"2c093b1619555ca6dbb2b91ccb7e3ada144c7a0e91c3b4dfd618bdb5a009f409"}
Nov 01 07:41:00.690701 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:00.690646    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg" podStartSLOduration=1.6906322409999999 podStartE2EDuration="1.690632241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:41:00.639435065 +0000 UTC m=+5472.207241895" watchObservedRunningTime="2026-11-01 07:41:00.690632241 +0000 UTC m=+5472.258439116"
Nov 01 07:41:01.028328 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:41:01.028301    4468 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fb1ef2_fc7b_4e5f_8728_2dbb762ddbf0.slice/crio-conmon-64dd20bd01a93d7f29c03f87d0eccc2cff638666e213a01e4683476a86d598ef.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 07:41:01.535212 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:01.535184    4468 generic.go:356] "Generic (PLEG): container finished" podID="08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" containerID="64dd20bd01a93d7f29c03f87d0eccc2cff638666e213a01e4683476a86d598ef" exitCode=0
Nov 01 07:41:01.535622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:01.535220    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg" event={"ID":"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0","Type":"ContainerDied","Data":"64dd20bd01a93d7f29c03f87d0eccc2cff638666e213a01e4683476a86d598ef"}
Nov 01 07:41:02.674155 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.674132    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:41:02.738732 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.738709    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca\") pod \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") "
Nov 01 07:41:02.738851 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.738791    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca\") pod \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") "
Nov 01 07:41:02.738893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.738858    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628\" (UniqueName: \"kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628\") pod \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\" (UID: \"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0\") "
Nov 01 07:41:02.739145 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.739115    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca" pod "08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" (UID: "08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:41:02.740950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.740929    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca" pod "08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" (UID: "08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:41:02.741052 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.740980    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628" pod "08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" (UID: "08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0"). InnerVolumeSpecName "kube-api-access-qs628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:41:02.840374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.840343    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs628\" (UniqueName: \"kubernetes.io/projected/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-kube-api-access-qs628\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:41:02.840374 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.840370    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:41:02.840576 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:02.840382    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:41:02.946212 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:41:02.946180    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:41:03.547675 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:03.547643    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg"
Nov 01 07:41:03.547920 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:03.547678    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c764ac-xphqg" event={"ID":"08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0","Type":"ContainerDied","Data":"2c093b1619555ca6dbb2b91ccb7e3ada144c7a0e91c3b4dfd618bdb5a009f409"}
Nov 01 07:41:03.547920 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:03.547723    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c093b1619555ca6dbb2b91ccb7e3ada144c7a0e91c3b4dfd618bdb5a009f409"
Nov 01 07:41:12.301904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:12.301875    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:41:24.044523 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:24.044486    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"]
Nov 01 07:41:24.094768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:24.094731    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-m6hdf"]
Nov 01 07:41:24.313839 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:24.313805    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:41:24.589781 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:24.589704    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59499c9d-fbad-4e7e-b5ba-6351a767b110" path="/var/lib/kubelet/pods/59499c9d-fbad-4e7e-b5ba-6351a767b110/volumes"
Nov 01 07:41:32.682910 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.682877    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"]
Nov 01 07:41:32.683325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.683279    4468 cpu_manager.go:414] "containerMap: removing container" podUID="08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" containerName="revoke-keys"
Nov 01 07:41:32.683325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.683290    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" containerName="revoke-keys"
Nov 01 07:41:32.683404 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.683368    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0" containerName="revoke-keys"
Nov 01 07:41:32.686736 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.686716    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.741460 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.741433    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:41:32.791396 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.791358    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:41:32.791593 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.791398    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:41:32.791593 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.791442    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:41:32.793950 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.793931    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"]
Nov 01 07:41:32.809359 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.809336    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.809523 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.809388    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.809582 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.809533    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c597v\" (UniqueName: \"kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.910138 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.910102    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.910362 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.910172    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c597v\" (UniqueName: \"kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.910362 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.910236    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.910862 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.910837    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.912660 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.912638    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.947510 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:41:32.947394    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:41:32.967077 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.967049    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c597v\" (UniqueName: \"kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v\") pod \"maas-api-revoke-sub-1b539e3d-5twbw\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:32.999093 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:32.999056    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:33.368903 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:33.368882    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"]
Nov 01 07:41:33.370673 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:41:33.370641    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9be1308_58ab_4573_ba9b_f79d4132466e.slice/crio-c8a6995a21e1896570f32eed2bc284da6c33c01b4ebab99a821659b167cccd31 WatchSource:0}: Error finding container c8a6995a21e1896570f32eed2bc284da6c33c01b4ebab99a821659b167cccd31: Status 404 returned error can't find the container with id c8a6995a21e1896570f32eed2bc284da6c33c01b4ebab99a821659b167cccd31
Nov 01 07:41:33.734016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:33.733925    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw" event={"ID":"c9be1308-58ab-4573-ba9b-f79d4132466e","Type":"ContainerStarted","Data":"e4be42ed9937ab9f75f56ea1248bfa8364482d2adcd6e72276762574387400b3"}
Nov 01 07:41:33.734016 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:33.733987    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw" event={"ID":"c9be1308-58ab-4573-ba9b-f79d4132466e","Type":"ContainerStarted","Data":"c8a6995a21e1896570f32eed2bc284da6c33c01b4ebab99a821659b167cccd31"}
Nov 01 07:41:33.849249 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:33.849197    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw" podStartSLOduration=1.849183423 podStartE2EDuration="1.849183423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:41:33.847633785 +0000 UTC m=+5505.415440616" watchObservedRunningTime="2026-11-01 07:41:33.849183423 +0000 UTC m=+5505.416990300"
Nov 01 07:41:34.740919 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:34.740883    4468 generic.go:356] "Generic (PLEG): container finished" podID="c9be1308-58ab-4573-ba9b-f79d4132466e" containerID="e4be42ed9937ab9f75f56ea1248bfa8364482d2adcd6e72276762574387400b3" exitCode=0
Nov 01 07:41:34.741353 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:34.740952    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw" event={"ID":"c9be1308-58ab-4573-ba9b-f79d4132466e","Type":"ContainerDied","Data":"e4be42ed9937ab9f75f56ea1248bfa8364482d2adcd6e72276762574387400b3"}
Nov 01 07:41:35.043279 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:35.043195    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"]
Nov 01 07:41:35.045303 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:35.045277    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zwzml"]
Nov 01 07:41:35.873777 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:35.873754    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:36.039495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.039398    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v\" (UniqueName: \"kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v\") pod \"c9be1308-58ab-4573-ba9b-f79d4132466e\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") "
Nov 01 07:41:36.039673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.039516    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca\") pod \"c9be1308-58ab-4573-ba9b-f79d4132466e\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") "
Nov 01 07:41:36.039673 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.039541    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca\") pod \"c9be1308-58ab-4573-ba9b-f79d4132466e\" (UID: \"c9be1308-58ab-4573-ba9b-f79d4132466e\") "
Nov 01 07:41:36.039936 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.039915    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca" pod "c9be1308-58ab-4573-ba9b-f79d4132466e" (UID: "c9be1308-58ab-4573-ba9b-f79d4132466e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:41:36.041668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.041643    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v" pod "c9be1308-58ab-4573-ba9b-f79d4132466e" (UID: "c9be1308-58ab-4573-ba9b-f79d4132466e"). InnerVolumeSpecName "kube-api-access-c597v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:41:36.041772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.041714    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca" pod "c9be1308-58ab-4573-ba9b-f79d4132466e" (UID: "c9be1308-58ab-4573-ba9b-f79d4132466e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:41:36.140735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.140700    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:41:36.140735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.140731    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be1308-58ab-4573-ba9b-f79d4132466e-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:41:36.140940 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.140745    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c597v\" (UniqueName: \"kubernetes.io/projected/c9be1308-58ab-4573-ba9b-f79d4132466e-kube-api-access-c597v\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:41:36.588977 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.588942    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9b1601-e7ea-4f36-895f-1cae56c8759a" path="/var/lib/kubelet/pods/3f9b1601-e7ea-4f36-895f-1cae56c8759a/volumes"
Nov 01 07:41:36.752084 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.752058    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw"
Nov 01 07:41:36.752259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.752058    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b539e3d-5twbw" event={"ID":"c9be1308-58ab-4573-ba9b-f79d4132466e","Type":"ContainerDied","Data":"c8a6995a21e1896570f32eed2bc284da6c33c01b4ebab99a821659b167cccd31"}
Nov 01 07:41:36.752259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:36.752179    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8a6995a21e1896570f32eed2bc284da6c33c01b4ebab99a821659b167cccd31"
Nov 01 07:41:45.504434 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:45.504400    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:41:53.337491 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:53.334325    4468 scope.go:122] "RemoveContainer" containerID="01045aab7a1bf060efb2d3eaf33d5c5e715a721b9b8af853802c046012e707e0"
Nov 01 07:41:53.368990 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:53.368612    4468 scope.go:122] "RemoveContainer" containerID="cd9d030a138c37ef6064192adefd7ed45748be730ad7ecb2f81b5861be5ea76c"
Nov 01 07:41:53.383107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:53.383007    4468 scope.go:122] "RemoveContainer" containerID="ed9187fba29a024dca29dabc45689f49106e1614fdfcbc2fc4b5867236d65969"
Nov 01 07:41:53.391195 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:53.391178    4468 scope.go:122] "RemoveContainer" containerID="c27af9a903e1092036c1a3ad3ac1e2071655e3855d97d1c4f5f1db2d236954fd"
Nov 01 07:41:57.898064 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:41:57.898031    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:42:00.815708 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:00.815663    4468 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-pn2tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:42:00.816368 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:00.816331    4468 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-pn2tc" podUID="c3026d10-479c-4eaa-a18f-d2a83a7b508d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:42:02.946213 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:42:02.946176    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:42:17.462958 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.462921    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"]
Nov 01 07:42:17.465518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.463476    4468 cpu_manager.go:414] "containerMap: removing container" podUID="c9be1308-58ab-4573-ba9b-f79d4132466e" containerName="revoke-keys"
Nov 01 07:42:17.465518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.463491    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9be1308-58ab-4573-ba9b-f79d4132466e" containerName="revoke-keys"
Nov 01 07:42:17.465518 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.463564    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9be1308-58ab-4573-ba9b-f79d4132466e" containerName="revoke-keys"
Nov 01 07:42:17.466493 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.466455    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.531074 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.531033    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlwm\" (UniqueName: \"kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.531280 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.531117    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.531360 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.531300    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.570115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.570066    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:42:17.570115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.570089    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:42:17.570346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.570243    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:42:17.570346 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.570286    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:42:17.572447 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.572426    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"]
Nov 01 07:42:17.632529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.632487    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.632721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.632574    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.632721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.632660    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlwm\" (UniqueName: \"kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.633158 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.633128    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.635128 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.635104    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.689357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.689318    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlwm\" (UniqueName: \"kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm\") pod \"maas-api-revoke-sub-6817c72a-lsch2\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:17.782364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:17.782258    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:18.151337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:18.151314    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"]
Nov 01 07:42:18.153235 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:42:18.153209    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2392c587_6d80_498e_8c43_a3c50c1c2b09.slice/crio-e005b21179ea7b9e3614e70e1ec439a2c3a9c29f6dcdfd54ce82c52f6d5c9873 WatchSource:0}: Error finding container e005b21179ea7b9e3614e70e1ec439a2c3a9c29f6dcdfd54ce82c52f6d5c9873: Status 404 returned error can't find the container with id e005b21179ea7b9e3614e70e1ec439a2c3a9c29f6dcdfd54ce82c52f6d5c9873
Nov 01 07:42:18.987768 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:18.987735    4468 generic.go:356] "Generic (PLEG): container finished" podID="2392c587-6d80-498e-8c43-a3c50c1c2b09" containerID="e112719eac3d9e82c33ff61cf36953e557f8895e542c3ff8127278ced998c72f" exitCode=0
Nov 01 07:42:18.988169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:18.987819    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2" event={"ID":"2392c587-6d80-498e-8c43-a3c50c1c2b09","Type":"ContainerDied","Data":"e112719eac3d9e82c33ff61cf36953e557f8895e542c3ff8127278ced998c72f"}
Nov 01 07:42:18.988169 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:18.987877    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2" event={"ID":"2392c587-6d80-498e-8c43-a3c50c1c2b09","Type":"ContainerStarted","Data":"e005b21179ea7b9e3614e70e1ec439a2c3a9c29f6dcdfd54ce82c52f6d5c9873"}
Nov 01 07:42:20.133059 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.133028    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:20.261120 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.261030    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm\" (UniqueName: \"kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm\") pod \"2392c587-6d80-498e-8c43-a3c50c1c2b09\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") "
Nov 01 07:42:20.261312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.261175    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca\") pod \"2392c587-6d80-498e-8c43-a3c50c1c2b09\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") "
Nov 01 07:42:20.261312 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.261279    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca\") pod \"2392c587-6d80-498e-8c43-a3c50c1c2b09\" (UID: \"2392c587-6d80-498e-8c43-a3c50c1c2b09\") "
Nov 01 07:42:20.261565 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.261537    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca" pod "2392c587-6d80-498e-8c43-a3c50c1c2b09" (UID: "2392c587-6d80-498e-8c43-a3c50c1c2b09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:42:20.261690 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.261663    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:42:20.263392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.263367    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca" pod "2392c587-6d80-498e-8c43-a3c50c1c2b09" (UID: "2392c587-6d80-498e-8c43-a3c50c1c2b09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:42:20.263526 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.263433    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm" pod "2392c587-6d80-498e-8c43-a3c50c1c2b09" (UID: "2392c587-6d80-498e-8c43-a3c50c1c2b09"). InnerVolumeSpecName "kube-api-access-6zlwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:42:20.362955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.362910    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2392c587-6d80-498e-8c43-a3c50c1c2b09-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:42:20.362955 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:20.362945    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zlwm\" (UniqueName: \"kubernetes.io/projected/2392c587-6d80-498e-8c43-a3c50c1c2b09-kube-api-access-6zlwm\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:42:21.000734 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:21.000658    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2"
Nov 01 07:42:21.000734 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:21.000705    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6817c72a-lsch2" event={"ID":"2392c587-6d80-498e-8c43-a3c50c1c2b09","Type":"ContainerDied","Data":"e005b21179ea7b9e3614e70e1ec439a2c3a9c29f6dcdfd54ce82c52f6d5c9873"}
Nov 01 07:42:21.000931 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:21.000773    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e005b21179ea7b9e3614e70e1ec439a2c3a9c29f6dcdfd54ce82c52f6d5c9873"
Nov 01 07:42:28.203887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.203848    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"]
Nov 01 07:42:28.204379 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.204364    4468 cpu_manager.go:414] "containerMap: removing container" podUID="2392c587-6d80-498e-8c43-a3c50c1c2b09" containerName="revoke-keys"
Nov 01 07:42:28.204438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.204381    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2392c587-6d80-498e-8c43-a3c50c1c2b09" containerName="revoke-keys"
Nov 01 07:42:28.204495 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.204479    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="2392c587-6d80-498e-8c43-a3c50c1c2b09" containerName="revoke-keys"
Nov 01 07:42:28.207744 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.207728    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.311961 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.311916    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:42:28.311961 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.311935    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:42:28.312240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.311980    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:42:28.312240 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.312009    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:42:28.314220 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.314199    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"]
Nov 01 07:42:28.347669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.347631    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.347933 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.347690    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.347933 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.347778    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrhg\" (UniqueName: \"kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.449484 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.449440    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.449677 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.449557    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.449677 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.449601    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrhg\" (UniqueName: \"kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.450242 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.450214    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.452086 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.452065    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.506755 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.506681    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrhg\" (UniqueName: \"kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg\") pod \"maas-api-revoke-sub-2a31f0ef-d55js\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.519998 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.519965    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:28.890854 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:28.890823    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"]
Nov 01 07:42:28.893637 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:42:28.893606    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95883de1_7c04_4770_b02d_f3a548acc532.slice/crio-a2cbf8874382eca8ad4d73c864d002b126c04785bd6cb389d35ed69e84259235 WatchSource:0}: Error finding container a2cbf8874382eca8ad4d73c864d002b126c04785bd6cb389d35ed69e84259235: Status 404 returned error can't find the container with id a2cbf8874382eca8ad4d73c864d002b126c04785bd6cb389d35ed69e84259235
Nov 01 07:42:29.042089 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:29.042056    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js" event={"ID":"95883de1-7c04-4770-b02d-f3a548acc532","Type":"ContainerStarted","Data":"a2cbf8874382eca8ad4d73c864d002b126c04785bd6cb389d35ed69e84259235"}
Nov 01 07:42:30.048737 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:30.048702    4468 generic.go:356] "Generic (PLEG): container finished" podID="95883de1-7c04-4770-b02d-f3a548acc532" containerID="e8e74b619283fbfa4fc7670e00426d841f345de03392b779178e93ea34fb692d" exitCode=0
Nov 01 07:42:30.049146 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:30.048788    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js" event={"ID":"95883de1-7c04-4770-b02d-f3a548acc532","Type":"ContainerDied","Data":"e8e74b619283fbfa4fc7670e00426d841f345de03392b779178e93ea34fb692d"}
Nov 01 07:42:30.304569 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:30.304489    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:42:31.187664 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.187633    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:31.277525 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.277495    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg\" (UniqueName: \"kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg\") pod \"95883de1-7c04-4770-b02d-f3a548acc532\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") "
Nov 01 07:42:31.277705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.277644    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca\") pod \"95883de1-7c04-4770-b02d-f3a548acc532\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") "
Nov 01 07:42:31.277705 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.277668    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca\") pod \"95883de1-7c04-4770-b02d-f3a548acc532\" (UID: \"95883de1-7c04-4770-b02d-f3a548acc532\") "
Nov 01 07:42:31.278057 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.278032    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca" pod "95883de1-7c04-4770-b02d-f3a548acc532" (UID: "95883de1-7c04-4770-b02d-f3a548acc532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:42:31.279760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.279733    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg" pod "95883de1-7c04-4770-b02d-f3a548acc532" (UID: "95883de1-7c04-4770-b02d-f3a548acc532"). InnerVolumeSpecName "kube-api-access-zlrhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:42:31.279760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.279731    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca" pod "95883de1-7c04-4770-b02d-f3a548acc532" (UID: "95883de1-7c04-4770-b02d-f3a548acc532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:42:31.378731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.378646    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:42:31.378731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.378680    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95883de1-7c04-4770-b02d-f3a548acc532-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:42:31.378731 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:31.378691    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrhg\" (UniqueName: \"kubernetes.io/projected/95883de1-7c04-4770-b02d-f3a548acc532-kube-api-access-zlrhg\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:42:32.061329 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:32.061299    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js"
Nov 01 07:42:32.061531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:32.061341    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a31f0ef-d55js" event={"ID":"95883de1-7c04-4770-b02d-f3a548acc532","Type":"ContainerDied","Data":"a2cbf8874382eca8ad4d73c864d002b126c04785bd6cb389d35ed69e84259235"}
Nov 01 07:42:32.061531 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:32.061391    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2cbf8874382eca8ad4d73c864d002b126c04785bd6cb389d35ed69e84259235"
Nov 01 07:42:32.947275 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:42:32.947240    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:42:40.985181 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:40.985146    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:42:53.315342 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:53.315304    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:42:53.499853 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:53.499818    4468 scope.go:122] "RemoveContainer" containerID="cc15196f9fa4956d457d33bbbabf02f2c435381c844aee6f54c06632ea140687"
Nov 01 07:42:53.508623 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:53.508598    4468 scope.go:122] "RemoveContainer" containerID="c81e9454409fb92a4580f42cbb8c17ac887e7292b77af3c431dbd717bd836996"
Nov 01 07:42:53.516527 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:42:53.516507    4468 scope.go:122] "RemoveContainer" containerID="f79f8157f75afd8bdb75fa0b77493f2ff585da6ff18563aa467ec8de698132a8"
Nov 01 07:43:02.946204 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:43:02.946165    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:43:05.693238 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:05.693204    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:43:16.177337 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.177297    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"]
Nov 01 07:43:16.177804 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.177768    4468 cpu_manager.go:414] "containerMap: removing container" podUID="95883de1-7c04-4770-b02d-f3a548acc532" containerName="revoke-keys"
Nov 01 07:43:16.177804 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.177782    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95883de1-7c04-4770-b02d-f3a548acc532" containerName="revoke-keys"
Nov 01 07:43:16.177895 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.177883    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="95883de1-7c04-4770-b02d-f3a548acc532" containerName="revoke-keys"
Nov 01 07:43:16.181262 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.181239    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.240103 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.240068    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"]
Nov 01 07:43:16.247685 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.247660    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.247833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.247702    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.247833 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.247722    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzpw\" (UniqueName: \"kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.285111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.285068    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:43:16.285111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.285100    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:43:16.285111 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.285078    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:43:16.285364 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.285172    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:43:16.348741 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.348702    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.348948 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.348752    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.348948 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.348778    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzpw\" (UniqueName: \"kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.349370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.349349    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.351273 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.351247    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.405881 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.405844    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzpw\" (UniqueName: \"kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw\") pod \"maas-api-revoke-sub-fe7f029a-7tdrk\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.497259 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.497171    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:16.868182 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:16.868154    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"]
Nov 01 07:43:16.870800 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:43:16.870772    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c820277_6384_42e7_8a1d_ceda90a74870.slice/crio-5a7acc09b9b02ac877d9a6c591c853eb243d66bda64ddc0c424606f68c50962e WatchSource:0}: Error finding container 5a7acc09b9b02ac877d9a6c591c853eb243d66bda64ddc0c424606f68c50962e: Status 404 returned error can't find the container with id 5a7acc09b9b02ac877d9a6c591c853eb243d66bda64ddc0c424606f68c50962e
Nov 01 07:43:17.418839 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:17.418792    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk" event={"ID":"0c820277-6384-42e7-8a1d-ceda90a74870","Type":"ContainerStarted","Data":"6dfddfc865fc7a6a1f57da014ecbdedde4968268662f406b51df2f29227a6775"}
Nov 01 07:43:17.419284 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:17.418849    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk" event={"ID":"0c820277-6384-42e7-8a1d-ceda90a74870","Type":"ContainerStarted","Data":"5a7acc09b9b02ac877d9a6c591c853eb243d66bda64ddc0c424606f68c50962e"}
Nov 01 07:43:17.533367 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:17.533310    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk" podStartSLOduration=1.533295151 podStartE2EDuration="1.533295151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:43:17.531613595 +0000 UTC m=+5609.099420424" watchObservedRunningTime="2026-11-01 07:43:17.533295151 +0000 UTC m=+5609.101101979"
Nov 01 07:43:18.425813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:18.425777    4468 generic.go:356] "Generic (PLEG): container finished" podID="0c820277-6384-42e7-8a1d-ceda90a74870" containerID="6dfddfc865fc7a6a1f57da014ecbdedde4968268662f406b51df2f29227a6775" exitCode=0
Nov 01 07:43:18.426211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:18.425866    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk" event={"ID":"0c820277-6384-42e7-8a1d-ceda90a74870","Type":"ContainerDied","Data":"6dfddfc865fc7a6a1f57da014ecbdedde4968268662f406b51df2f29227a6775"}
Nov 01 07:43:19.566438 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.566405    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:19.685107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.685030    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca\") pod \"0c820277-6384-42e7-8a1d-ceda90a74870\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") "
Nov 01 07:43:19.685272 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.685194    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw\" (UniqueName: \"kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw\") pod \"0c820277-6384-42e7-8a1d-ceda90a74870\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") "
Nov 01 07:43:19.685340 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.685276    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca\") pod \"0c820277-6384-42e7-8a1d-ceda90a74870\" (UID: \"0c820277-6384-42e7-8a1d-ceda90a74870\") "
Nov 01 07:43:19.685401 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.685378    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca" pod "0c820277-6384-42e7-8a1d-ceda90a74870" (UID: "0c820277-6384-42e7-8a1d-ceda90a74870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:43:19.685783 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.685760    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:43:19.687641 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.687611    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw" pod "0c820277-6384-42e7-8a1d-ceda90a74870" (UID: "0c820277-6384-42e7-8a1d-ceda90a74870"). InnerVolumeSpecName "kube-api-access-cxzpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:43:19.687760 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.687663    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca" pod "0c820277-6384-42e7-8a1d-ceda90a74870" (UID: "0c820277-6384-42e7-8a1d-ceda90a74870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:43:19.787172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.787136    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c820277-6384-42e7-8a1d-ceda90a74870-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:43:19.787172 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:19.787167    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzpw\" (UniqueName: \"kubernetes.io/projected/0c820277-6384-42e7-8a1d-ceda90a74870-kube-api-access-cxzpw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:43:20.436821 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:20.436791    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk"
Nov 01 07:43:20.437031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:20.436858    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7f029a-7tdrk" event={"ID":"0c820277-6384-42e7-8a1d-ceda90a74870","Type":"ContainerDied","Data":"5a7acc09b9b02ac877d9a6c591c853eb243d66bda64ddc0c424606f68c50962e"}
Nov 01 07:43:20.437031 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:20.436909    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a7acc09b9b02ac877d9a6c591c853eb243d66bda64ddc0c424606f68c50962e"
Nov 01 07:43:26.044183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.044142    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"]
Nov 01 07:43:26.046772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.046746    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baece1ba-rpntn"]
Nov 01 07:43:26.588684 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.588650    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e78fe45-8621-4aac-be12-3006a055454f" path="/var/lib/kubelet/pods/2e78fe45-8621-4aac-be12-3006a055454f/volumes"
Nov 01 07:43:26.967647 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.967567    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"]
Nov 01 07:43:26.967984 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.967972    4468 cpu_manager.go:414] "containerMap: removing container" podUID="0c820277-6384-42e7-8a1d-ceda90a74870" containerName="revoke-keys"
Nov 01 07:43:26.968024 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.967985    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c820277-6384-42e7-8a1d-ceda90a74870" containerName="revoke-keys"
Nov 01 07:43:26.968092 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.968073    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c820277-6384-42e7-8a1d-ceda90a74870" containerName="revoke-keys"
Nov 01 07:43:26.971302 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:26.971286    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.029047 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.029014    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"]
Nov 01 07:43:27.062942 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.062903    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6rw\" (UniqueName: \"kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.063345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.062959    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.063345 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.063136    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.075622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.075589    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:43:27.075622 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.075593    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:43:27.075802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.075632    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:43:27.075802 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.075586    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 07:43:27.164539 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.164511    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.164764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.164616    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6rw\" (UniqueName: \"kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.164764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.164644    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.165365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.165338    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.167048 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.167030    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.221083 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.221011    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6rw\" (UniqueName: \"kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw\") pod \"maas-api-revoke-sub-dc9e9c6c-zmxm5\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.283293 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.283257    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:27.655478 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:27.655433    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"]
Nov 01 07:43:27.657201 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:43:27.657170    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf772ef_5aba_446a_8321_f59b11f190bf.slice/crio-4186a30c9b4e4ab2a2969bf7c86963b487d971c8d09839fc686bd000b1cee4e2 WatchSource:0}: Error finding container 4186a30c9b4e4ab2a2969bf7c86963b487d971c8d09839fc686bd000b1cee4e2: Status 404 returned error can't find the container with id 4186a30c9b4e4ab2a2969bf7c86963b487d971c8d09839fc686bd000b1cee4e2
Nov 01 07:43:28.478268 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:28.478235    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5" event={"ID":"fdf772ef-5aba-446a-8321-f59b11f190bf","Type":"ContainerStarted","Data":"a04e5e2384296c95e547db88b28b24c8e80b0f2ecf010fe623b20b40eee0adf9"}
Nov 01 07:43:28.478696 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:28.478296    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5" event={"ID":"fdf772ef-5aba-446a-8321-f59b11f190bf","Type":"ContainerStarted","Data":"4186a30c9b4e4ab2a2969bf7c86963b487d971c8d09839fc686bd000b1cee4e2"}
Nov 01 07:43:28.592579 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:28.592514    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5" podStartSLOduration=2.592497581 podStartE2EDuration="2.592497581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:43:28.592015907 +0000 UTC m=+5620.159822736" watchObservedRunningTime="2026-11-01 07:43:28.592497581 +0000 UTC m=+5620.160304411"
Nov 01 07:43:29.009907 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:29.009865    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:43:29.486631 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:29.486590    4468 generic.go:356] "Generic (PLEG): container finished" podID="fdf772ef-5aba-446a-8321-f59b11f190bf" containerID="a04e5e2384296c95e547db88b28b24c8e80b0f2ecf010fe623b20b40eee0adf9" exitCode=0
Nov 01 07:43:29.487183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:29.486688    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5" event={"ID":"fdf772ef-5aba-446a-8321-f59b11f190bf","Type":"ContainerDied","Data":"a04e5e2384296c95e547db88b28b24c8e80b0f2ecf010fe623b20b40eee0adf9"}
Nov 01 07:43:30.958503 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:30.957745    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:31.042006 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.041969    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw\" (UniqueName: \"kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw\") pod \"fdf772ef-5aba-446a-8321-f59b11f190bf\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") "
Nov 01 07:43:31.042300 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.042277    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca\") pod \"fdf772ef-5aba-446a-8321-f59b11f190bf\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") "
Nov 01 07:43:31.042454 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.042438    4468 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca\") pod \"fdf772ef-5aba-446a-8321-f59b11f190bf\" (UID: \"fdf772ef-5aba-446a-8321-f59b11f190bf\") "
Nov 01 07:43:31.043383 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.043359    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca" pod "fdf772ef-5aba-446a-8321-f59b11f190bf" (UID: "fdf772ef-5aba-446a-8321-f59b11f190bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:43:31.061945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.059926    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca" pod "fdf772ef-5aba-446a-8321-f59b11f190bf" (UID: "fdf772ef-5aba-446a-8321-f59b11f190bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:43:31.061945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.060047    4468 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw" pod "fdf772ef-5aba-446a-8321-f59b11f190bf" (UID: "fdf772ef-5aba-446a-8321-f59b11f190bf"). InnerVolumeSpecName "kube-api-access-bt6rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:43:31.174764 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.174399    4468 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6rw\" (UniqueName: \"kubernetes.io/projected/fdf772ef-5aba-446a-8321-f59b11f190bf-kube-api-access-bt6rw\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:43:31.175005 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.174985    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-xks-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:43:31.175141 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.175126    4468 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf772ef-5aba-446a-8321-f59b11f190bf-maas-api-openshift-ca\") on node \"ip-10-0-138-21.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:43:31.498229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.498154    4468 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5"
Nov 01 07:43:31.498378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.498157    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9e9c6c-zmxm5" event={"ID":"fdf772ef-5aba-446a-8321-f59b11f190bf","Type":"ContainerDied","Data":"4186a30c9b4e4ab2a2969bf7c86963b487d971c8d09839fc686bd000b1cee4e2"}
Nov 01 07:43:31.498378 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:31.498265    4468 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4186a30c9b4e4ab2a2969bf7c86963b487d971c8d09839fc686bd000b1cee4e2"
Nov 01 07:43:32.946341 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:43:32.946302    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:43:39.659107 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:43:39.659067    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-b57dc"]
Nov 01 07:44:02.946777 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:44:02.946741    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:44:06.947993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:06.947959    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pwf6c_409da98c-561b-4be9-99cc-6535a33b667f/manager/0.log"
Nov 01 07:44:07.776887 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:07.776857    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d475bcc9-cs8xr_b3ba3540-a6bb-464f-89b1-0288581d1d47/manager/0.log"
Nov 01 07:44:08.779055 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:08.779024    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xg8kb_f3ec2313-e247-4d2a-8d15-465ba35bd9f3/proxy/0.log"
Nov 01 07:44:10.484868 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:10.484836    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1b539e3d-5twbw_c9be1308-58ab-4573-ba9b-f79d4132466e/revoke-keys/0.log"
Nov 01 07:44:10.706109 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:10.706077    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2a31f0ef-d55js_95883de1-7c04-4770-b02d-f3a548acc532/revoke-keys/0.log"
Nov 01 07:44:10.921668 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:10.921636    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-pwkpj_34c60c3f-90ca-4c68-89a0-3d63914ca408/revoke-keys/0.log"
Nov 01 07:44:11.130722 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:11.130693    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-47c764ac-xphqg_08fb1ef2-fc7b-4e5f-8728-2dbb762ddbf0/revoke-keys/0.log"
Nov 01 07:44:11.332455 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:11.332427    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5f02c688-dgj9q_5d0fc632-70fa-41bb-9659-a377652bff32/revoke-keys/0.log"
Nov 01 07:44:11.559005 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:11.558975    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6817c72a-lsch2_2392c587-6d80-498e-8c43-a3c50c1c2b09/revoke-keys/0.log"
Nov 01 07:44:11.778456 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:11.778353    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-p9l8r_f6baa1a7-5464-40e9-932a-6c1683c4f252/revoke-keys/0.log"
Nov 01 07:44:12.004408 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:12.004377    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dc9e9c6c-zmxm5_fdf772ef-5aba-446a-8321-f59b11f190bf/revoke-keys/0.log"
Nov 01 07:44:12.235018 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:12.234986    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fe7f029a-7tdrk_0c820277-6384-42e7-8a1d-ceda90a74870/revoke-keys/0.log"
Nov 01 07:44:16.029721 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:16.029692    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-b57dc_81cbf762-f2c8-44ae-b2a5-e1a8e7166be8/limitador/0.log"
Nov 01 07:44:18.059847 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:18.059816    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cbf89484-cmqtk_c2673628-ba47-467c-a366-ee01f7c3036d/kube-auth-proxy/0.log"
Nov 01 07:44:19.518892 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:19.518844    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hjj86_8dfde8ce-8829-4310-a77e-0e64d77c0c45/istio-proxy/0.log"
Nov 01 07:44:20.116265 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:20.116217    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-pn2tc_c3026d10-479c-4eaa-a18f-d2a83a7b508d/istio-proxy/0.log"
Nov 01 07:44:20.396189 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:20.396101    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-qb2mm_a7adc820-4dbf-455b-ada4-9b655c77c786/istio-proxy/0.log"
Nov 01 07:44:23.994060 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:23.993978    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-gh8jf_1cbc55b4-a72b-493e-8cc8-78d4d1456b81/main/0.log"
Nov 01 07:44:24.542993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:24.542966    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-5bs7b_7df7649b-cb74-4980-92a2-890009c64658/main/0.log"
Nov 01 07:44:24.816805 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:24.816778    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6r8gx_b3a44132-55e7-4bd3-ad44-7c5317b44948/main/0.log"
Nov 01 07:44:25.036627 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:25.036581    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rf6nx_82915854-11e3-466d-a38e-2f235096f61c/main/0.log"
Nov 01 07:44:32.945975 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:44:32.945936    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:44:35.575986 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.575947    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s7pxj/must-gather-5n4ft"]
Nov 01 07:44:35.576601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.576447    4468 cpu_manager.go:414] "containerMap: removing container" podUID="fdf772ef-5aba-446a-8321-f59b11f190bf" containerName="revoke-keys"
Nov 01 07:44:35.576601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.576461    4468 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf772ef-5aba-446a-8321-f59b11f190bf" containerName="revoke-keys"
Nov 01 07:44:35.576601 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.576591    4468 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf772ef-5aba-446a-8321-f59b11f190bf" containerName="revoke-keys"
Nov 01 07:44:35.580033 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.580016    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.632211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.632167    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s7pxj\"/\"kube-root-ca.crt\""
Nov 01 07:44:35.632211 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.632175    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s7pxj\"/\"openshift-service-ca.crt\""
Nov 01 07:44:35.639766 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.639736    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/cfdc376a-b891-46c5-9715-8219f967dc3a-kube-api-access-rsdqn\") pod \"must-gather-5n4ft\" (UID: \"cfdc376a-b891-46c5-9715-8219f967dc3a\") " pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.639914 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.639889    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfdc376a-b891-46c5-9715-8219f967dc3a-must-gather-output\") pod \"must-gather-5n4ft\" (UID: \"cfdc376a-b891-46c5-9715-8219f967dc3a\") " pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.682250 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.682219    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s7pxj\"/\"default-dockercfg-ddg29\""
Nov 01 07:44:35.684616 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.684593    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s7pxj/must-gather-5n4ft"]
Nov 01 07:44:35.741178 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.741143    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/cfdc376a-b891-46c5-9715-8219f967dc3a-kube-api-access-rsdqn\") pod \"must-gather-5n4ft\" (UID: \"cfdc376a-b891-46c5-9715-8219f967dc3a\") " pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.741377 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.741243    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfdc376a-b891-46c5-9715-8219f967dc3a-must-gather-output\") pod \"must-gather-5n4ft\" (UID: \"cfdc376a-b891-46c5-9715-8219f967dc3a\") " pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.741609 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.741590    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cfdc376a-b891-46c5-9715-8219f967dc3a-must-gather-output\") pod \"must-gather-5n4ft\" (UID: \"cfdc376a-b891-46c5-9715-8219f967dc3a\") " pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.798116 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.798088    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/cfdc376a-b891-46c5-9715-8219f967dc3a-kube-api-access-rsdqn\") pod \"must-gather-5n4ft\" (UID: \"cfdc376a-b891-46c5-9715-8219f967dc3a\") " pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:35.900628 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:35.900548    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s7pxj/must-gather-5n4ft"
Nov 01 07:44:36.269827 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:36.269799    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s7pxj/must-gather-5n4ft"]
Nov 01 07:44:36.271687 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:44:36.271649    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfdc376a_b891_46c5_9715_8219f967dc3a.slice/crio-0e3965a64c5aaa83d28d2c9a0c6557d64632f5f707a5f33576a74272c9d8cd85 WatchSource:0}: Error finding container 0e3965a64c5aaa83d28d2c9a0c6557d64632f5f707a5f33576a74272c9d8cd85: Status 404 returned error can't find the container with id 0e3965a64c5aaa83d28d2c9a0c6557d64632f5f707a5f33576a74272c9d8cd85
Nov 01 07:44:36.273893 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:36.273877    4468 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 07:44:36.847370 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:36.847335    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7pxj/must-gather-5n4ft" event={"ID":"cfdc376a-b891-46c5-9715-8219f967dc3a","Type":"ContainerStarted","Data":"0e3965a64c5aaa83d28d2c9a0c6557d64632f5f707a5f33576a74272c9d8cd85"}
Nov 01 07:44:42.887820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:42.887724    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7pxj/must-gather-5n4ft" event={"ID":"cfdc376a-b891-46c5-9715-8219f967dc3a","Type":"ContainerStarted","Data":"b87864ad8c03c2a09e41a4391b2698be0debd2892963d27c62b555ad68fcb941"}
Nov 01 07:44:42.887820 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:42.887768    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7pxj/must-gather-5n4ft" event={"ID":"cfdc376a-b891-46c5-9715-8219f967dc3a","Type":"ContainerStarted","Data":"8f18ad49784a30ccf3fda345e8a083af77e9c35b96f71f39ed012f30fe503f5f"}
Nov 01 07:44:43.002529 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:43.002443    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s7pxj/must-gather-5n4ft" podStartSLOduration=1.704271638 podStartE2EDuration="8.00242175s" totalImagesPullingTime="6.298150112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:44:43.000328235 +0000 UTC m=+5694.568135086" watchObservedRunningTime="2026-11-01 07:44:43.00242175 +0000 UTC m=+5694.570228580"
Nov 01 07:44:45.806432 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:45.806394    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v48xf_8b62b472-60a7-442e-a38f-e7ca7b5172ec/global-pull-secret-syncer/0.log"
Nov 01 07:44:47.160621 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:47.160592    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pwf6c_409da98c-561b-4be9-99cc-6535a33b667f/manager/0.log"
Nov 01 07:44:48.195365 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:48.195327    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d475bcc9-cs8xr_b3ba3540-a6bb-464f-89b1-0288581d1d47/manager/0.log"
Nov 01 07:44:48.666122 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:48.666083    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w6l2n_b73c5c00-4399-4546-a2fe-7f58e086e5a8/konnectivity-agent/0.log"
Nov 01 07:44:49.484904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:49.484871    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-21.us-west-1.compute.internal_5a65f0fc8a51566f4a4d62ed9c49d84d/haproxy/0.log"
Nov 01 07:44:50.477669 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:50.477635    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xg8kb_f3ec2313-e247-4d2a-8d15-465ba35bd9f3/proxy/0.log"
Nov 01 07:44:52.044578 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:52.044540    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"]
Nov 01 07:44:52.094894 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:52.094853    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pwkpj"]
Nov 01 07:44:52.589158 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:52.589122    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c60c3f-90ca-4c68-89a0-3d63914ca408" path="/var/lib/kubelet/pods/34c60c3f-90ca-4c68-89a0-3d63914ca408/volumes"
Nov 01 07:44:53.550904 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:53.550872    4468 scope.go:122] "RemoveContainer" containerID="b252bb00720a858d535b46905dfc9b7863fd632c8871c025a05a565908f655af"
Nov 01 07:44:53.863988 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:53.863907    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pwf6c_409da98c-561b-4be9-99cc-6535a33b667f/manager/0.log"
Nov 01 07:44:54.288152 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.288074    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"]
Nov 01 07:44:54.295105 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.295081    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.348184 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.348138    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgfcw\" (UniqueName: \"kubernetes.io/projected/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-kube-api-access-sgfcw\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.348445 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.348418    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-podres\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.348608 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.348531    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-sys\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.348608 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.348603    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-proc\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.348733 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.348704    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-lib-modules\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.349680 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.349662    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"]
Nov 01 07:44:54.396772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.396745    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kd9k9\"/\"kube-root-ca.crt\""
Nov 01 07:44:54.396772 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.396749    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kd9k9\"/\"openshift-service-ca.crt\""
Nov 01 07:44:54.398126 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.398108    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kd9k9\"/\"default-dockercfg-hbgcf\""
Nov 01 07:44:54.450318 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450269    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-sys\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450379    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-proc\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450417    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-sys\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450455    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-lib-modules\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450534 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450524    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-proc\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450569    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-lib-modules\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450528    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfcw\" (UniqueName: \"kubernetes.io/projected/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-kube-api-access-sgfcw\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450631    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-podres\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.450773 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.450729    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-podres\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.507902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.507863    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfcw\" (UniqueName: \"kubernetes.io/projected/9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9-kube-api-access-sgfcw\") pod \"perf-node-gather-daemonset-rjfr2\" (UID: \"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.608709 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.608675    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:54.947357 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.947279    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d475bcc9-cs8xr_b3ba3540-a6bb-464f-89b1-0288581d1d47/manager/0.log"
Nov 01 07:44:54.979832 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:54.979807    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"]
Nov 01 07:44:54.981434 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:44:54.981408    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a5ae530_c43b_4a53_8a3f_4ea3dcdf17c9.slice/crio-3679911fdeb10f4e4670144a324336cb7a26e352ec7baf03ce88dfbb214c5881 WatchSource:0}: Error finding container 3679911fdeb10f4e4670144a324336cb7a26e352ec7baf03ce88dfbb214c5881: Status 404 returned error can't find the container with id 3679911fdeb10f4e4670144a324336cb7a26e352ec7baf03ce88dfbb214c5881
Nov 01 07:44:55.958632 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:55.958590    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2" event={"ID":"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9","Type":"ContainerStarted","Data":"a4071aa5a8aea1673683ce92d5d1ade2654bd0944f0de3c22a71eac1d64a4f53"}
Nov 01 07:44:55.959078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:55.958649    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2" event={"ID":"9a5ae530-c43b-4a53-8a3f-4ea3dcdf17c9","Type":"ContainerStarted","Data":"3679911fdeb10f4e4670144a324336cb7a26e352ec7baf03ce88dfbb214c5881"}
Nov 01 07:44:55.959078 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:55.958720    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:44:56.095188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:56.095135    4468 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2" podStartSLOduration=2.09511822 podStartE2EDuration="2.09511822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 07:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:44:56.092804745 +0000 UTC m=+5707.660611573" watchObservedRunningTime="2026-11-01 07:44:56.09511822 +0000 UTC m=+5707.662925050"
Nov 01 07:44:57.229170 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:57.229128    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xg8kb_f3ec2313-e247-4d2a-8d15-465ba35bd9f3/proxy/0.log"
Nov 01 07:44:57.229667 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:44:57.229241    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-b57dc_81cbf762-f2c8-44ae-b2a5-e1a8e7166be8/limitador/0.log"
Nov 01 07:45:00.161735 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.161582    4468 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"]
Nov 01 07:45:00.169597 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.169573    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.216023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.215971    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkcf\" (UniqueName: \"kubernetes.io/projected/69ca9e00-2475-4048-83f4-6fa5eeae29f6-kube-api-access-8nkcf\") pod \"maas-api-key-cleanup-29891985-prs7f\" (UID: \"69ca9e00-2475-4048-83f4-6fa5eeae29f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.216023 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.216029    4468 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ca9e00-2475-4048-83f4-6fa5eeae29f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891985-prs7f\" (UID: \"69ca9e00-2475-4048-83f4-6fa5eeae29f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.221884 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.221849    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"]
Nov 01 07:45:00.269148 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.269100    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 07:45:00.269148 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.269144    4468 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfqr4\""
Nov 01 07:45:00.271565 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.271534    4468 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 07:45:00.317297 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.317263    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkcf\" (UniqueName: \"kubernetes.io/projected/69ca9e00-2475-4048-83f4-6fa5eeae29f6-kube-api-access-8nkcf\") pod \"maas-api-key-cleanup-29891985-prs7f\" (UID: \"69ca9e00-2475-4048-83f4-6fa5eeae29f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.317297 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.317305    4468 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ca9e00-2475-4048-83f4-6fa5eeae29f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891985-prs7f\" (UID: \"69ca9e00-2475-4048-83f4-6fa5eeae29f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.318115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.318080    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ca9e00-2475-4048-83f4-6fa5eeae29f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891985-prs7f\" (UID: \"69ca9e00-2475-4048-83f4-6fa5eeae29f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.337813 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.337786    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/alertmanager/0.log"
Nov 01 07:45:00.374859 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.374819    4468 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkcf\" (UniqueName: \"kubernetes.io/projected/69ca9e00-2475-4048-83f4-6fa5eeae29f6-kube-api-access-8nkcf\") pod \"maas-api-key-cleanup-29891985-prs7f\" (UID: \"69ca9e00-2475-4048-83f4-6fa5eeae29f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.486902 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.486814    4468 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"
Nov 01 07:45:00.663163 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.663123    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pwf6c_409da98c-561b-4be9-99cc-6535a33b667f/manager/0.log"
Nov 01 07:45:00.869700 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.869673    4468 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f"]
Nov 01 07:45:00.871629 ip-10-0-138-21 kubenswrapper[4468]: W1101 07:45:00.871600    4468 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ca9e00_2475_4048_83f4_6fa5eeae29f6.slice/crio-013c041e57a3ce18642df526a74f0d5e3adf2825ff92d361ef81a7e410d13622 WatchSource:0}: Error finding container 013c041e57a3ce18642df526a74f0d5e3adf2825ff92d361ef81a7e410d13622: Status 404 returned error can't find the container with id 013c041e57a3ce18642df526a74f0d5e3adf2825ff92d361ef81a7e410d13622
Nov 01 07:45:00.949090 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.949058    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/config-reloader/0.log"
Nov 01 07:45:00.986301 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:00.986269    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f" event={"ID":"69ca9e00-2475-4048-83f4-6fa5eeae29f6","Type":"ContainerStarted","Data":"013c041e57a3ce18642df526a74f0d5e3adf2825ff92d361ef81a7e410d13622"}
Nov 01 07:45:01.362132 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.362094    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/kube-rbac-proxy-web/0.log"
Nov 01 07:45:01.776146 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.776112    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/kube-rbac-proxy/0.log"
Nov 01 07:45:01.787662 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.787628    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d475bcc9-cs8xr_b3ba3540-a6bb-464f-89b1-0288581d1d47/manager/0.log"
Nov 01 07:45:01.975325 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.975250    4468 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-rjfr2"
Nov 01 07:45:01.993305 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.993270    4468 generic.go:356] "Generic (PLEG): container finished" podID="69ca9e00-2475-4048-83f4-6fa5eeae29f6" containerID="83d2957814ab64e30ebf644a4efddbe7bbc3bcb98caa0ee1ea2705d17364b40d" exitCode=60
Nov 01 07:45:01.993520 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.993334    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f" event={"ID":"69ca9e00-2475-4048-83f4-6fa5eeae29f6","Type":"ContainerDied","Data":"83d2957814ab64e30ebf644a4efddbe7bbc3bcb98caa0ee1ea2705d17364b40d"}
Nov 01 07:45:01.993670 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:01.993654    4468 scope.go:122] "RemoveContainer" containerID="83d2957814ab64e30ebf644a4efddbe7bbc3bcb98caa0ee1ea2705d17364b40d"
Nov 01 07:45:02.204954 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:02.204920    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/kube-rbac-proxy-metric/0.log"
Nov 01 07:45:02.624823 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:02.624743    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/prom-label-proxy/0.log"
Nov 01 07:45:02.948521 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:45:02.948485    4468 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8k87" containerName="extractor"
Nov 01 07:45:03.001779 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:03.001744    4468 generic.go:356] "Generic (PLEG): container finished" podID="69ca9e00-2475-4048-83f4-6fa5eeae29f6" containerID="6060ca8a7dca8cade264943c244b48ecdd8d4f4c79b9dd5a3c84d6ef07402bb3" exitCode=60
Nov 01 07:45:03.001993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:03.001808    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f" event={"ID":"69ca9e00-2475-4048-83f4-6fa5eeae29f6","Type":"ContainerDied","Data":"6060ca8a7dca8cade264943c244b48ecdd8d4f4c79b9dd5a3c84d6ef07402bb3"}
Nov 01 07:45:03.001993 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:03.001856    4468 scope.go:122] "RemoveContainer" containerID="83d2957814ab64e30ebf644a4efddbe7bbc3bcb98caa0ee1ea2705d17364b40d"
Nov 01 07:45:03.002168 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:03.002152    4468 scope.go:122] "RemoveContainer" containerID="6060ca8a7dca8cade264943c244b48ecdd8d4f4c79b9dd5a3c84d6ef07402bb3"
Nov 01 07:45:03.002439 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:45:03.002418    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891985-prs7f_odh-ai-gateway-infra(69ca9e00-2475-4048-83f4-6fa5eeae29f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f" podUID="69ca9e00-2475-4048-83f4-6fa5eeae29f6"
Nov 01 07:45:03.039188 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:03.039115    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0100c589-3e24-459b-8baf-ac0337238eb6/init-config-reloader/0.log"
Nov 01 07:45:03.879945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:03.879914    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2rr5p_2a184847-b655-4c64-9c4a-4f10d6a50666/kube-state-metrics/0.log"
Nov 01 07:45:04.008569 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:04.008540    4468 scope.go:122] "RemoveContainer" containerID="6060ca8a7dca8cade264943c244b48ecdd8d4f4c79b9dd5a3c84d6ef07402bb3"
Nov 01 07:45:04.008832 ip-10-0-138-21 kubenswrapper[4468]: E1101 07:45:04.008808    4468 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891985-prs7f_odh-ai-gateway-infra(69ca9e00-2475-4048-83f4-6fa5eeae29f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f" podUID="69ca9e00-2475-4048-83f4-6fa5eeae29f6"
Nov 01 07:45:04.287113 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:04.287006    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2rr5p_2a184847-b655-4c64-9c4a-4f10d6a50666/kube-rbac-proxy-main/0.log"
Nov 01 07:45:04.524503 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:04.524454    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xg8kb_f3ec2313-e247-4d2a-8d15-465ba35bd9f3/proxy/0.log"
Nov 01 07:45:04.704183 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:04.704127    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2rr5p_2a184847-b655-4c64-9c4a-4f10d6a50666/kube-rbac-proxy-self/0.log"
Nov 01 07:45:05.121515 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:05.121489    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7687d8f685-7fxmj_88f65491-8bba-42dc-892d-3050d6e52db1/metrics-server/0.log"
Nov 01 07:45:05.537392 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:05.537312    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4ql2v_0ef38052-fddb-42b0-9740-f6b81a8a50e0/monitoring-plugin/0.log"
Nov 01 07:45:05.960458 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:05.960417    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fqht_4b9debf7-11c6-49c6-b89c-a9116021cee6/node-exporter/0.log"
Nov 01 07:45:06.373066 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:06.373038    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fqht_4b9debf7-11c6-49c6-b89c-a9116021cee6/kube-rbac-proxy/0.log"
Nov 01 07:45:06.786448 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:06.786357    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fqht_4b9debf7-11c6-49c6-b89c-a9116021cee6/init-textfile/0.log"
Nov 01 07:45:08.795255 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:08.795225    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cbf89484-cmqtk_c2673628-ba47-467c-a366-ee01f7c3036d/kube-auth-proxy/0.log"
Nov 01 07:45:09.723888 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:09.723857    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bgzgv_afbcf8d2-91c8-4732-a714-7ada04cbda4b/kube-rbac-proxy-main/0.log"
Nov 01 07:45:09.768707 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:09.768683    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hjj86_8dfde8ce-8829-4310-a77e-0e64d77c0c45/istio-proxy/0.log"
Nov 01 07:45:10.140015 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:10.139976    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bgzgv_afbcf8d2-91c8-4732-a714-7ada04cbda4b/kube-rbac-proxy-self/0.log"
Nov 01 07:45:10.197418 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:10.197377    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-pn2tc_c3026d10-479c-4eaa-a18f-d2a83a7b508d/istio-proxy/0.log"
Nov 01 07:45:10.409239 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:10.409164    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-qb2mm_a7adc820-4dbf-455b-ada4-9b655c77c786/istio-proxy/0.log"
Nov 01 07:45:10.555566 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:10.555538    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bgzgv_afbcf8d2-91c8-4732-a714-7ada04cbda4b/openshift-state-metrics/0.log"
Nov 01 07:45:10.979229 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:10.979202    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/prometheus/0.log"
Nov 01 07:45:11.391158 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:11.391127    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/config-reloader/0.log"
Nov 01 07:45:11.803948 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:11.803924    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/thanos-sidecar/0.log"
Nov 01 07:45:12.218209 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:12.218134    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/kube-rbac-proxy-web/0.log"
Nov 01 07:45:12.630676 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:12.630650    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/kube-rbac-proxy/0.log"
Nov 01 07:45:13.043543 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:13.043453    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/kube-rbac-proxy-thanos/0.log"
Nov 01 07:45:13.456945 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:13.456908    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1cc29dcb-6cd2-44f2-9e45-327b1a320940/init-config-reloader/0.log"
Nov 01 07:45:15.043362 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:15.043329    4468 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"]
Nov 01 07:45:15.093040 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:15.093001    4468 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-p9l8r"]
Nov 01 07:45:15.165482 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:15.165422    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-647855668f-9965p_0c6450cc-d4a7-48fb-ad27-93d76897bc71/telemeter-client/0.log"
Nov 01 07:45:15.577224 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:15.577195    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-647855668f-9965p_0c6450cc-d4a7-48fb-ad27-93d76897bc71/reload/0.log"
Nov 01 07:45:15.992011 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:15.991932    4468 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-647855668f-9965p_0c6450cc-d4a7-48fb-ad27-93d76897bc71/kube-rbac-proxy/0.log"
Nov 01 07:45:16.583441 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:16.583413    4468 scope.go:122] "RemoveContainer" containerID="6060ca8a7dca8cade264943c244b48ecdd8d4f4c79b9dd5a3c84d6ef07402bb3"
Nov 01 07:45:16.590008 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:16.589983    4468 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6baa1a7-5464-40e9-932a-6c1683c4f252" path="/var/lib/kubelet/pods/f6baa1a7-5464-40e9-932a-6c1683c4f252/volumes"
Nov 01 07:45:17.081115 ip-10-0-138-21 kubenswrapper[4468]: I1101 07:45:17.081080    4468 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891985-prs7f" event={"ID":"69ca9e00-2475-4048-83f4-6fa5eeae29f6","Type":"ContainerStarted","Data":"48a8f7d97f6a3729eb5a6f56a9a6fcd284a44abaffa4a39f0bde58e8fb04da00"}